Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] databricks_cluster_library resource #904

Closed
nfx opened this issue Nov 10, 2021 · 0 comments · Fixed by #971
Closed

[FEATURE] databricks_cluster_library resource #904

nfx opened this issue Nov 10, 2021 · 0 comments · Fixed by #971
Labels
feature New feature or request

Comments

@nfx
Copy link
Contributor

nfx commented Nov 10, 2021

Adding libraries to clusters may impose certain challenges mentioned in #820, #468, and #599. These issues become harder, when artifacts are installed outside of terraform workflow.

Scope of the issue is to modify databricks_cluster to ignore any changes to libraries, if they are either managed by new databricks_cluster_library resource or outside of TF workflow.

@nfx nfx added the feature New feature or request label Dec 11, 2021
nfx added a commit that referenced this issue Dec 15, 2021
```hcl
data "databricks_clusters" "all" {
}

resource "databricks_library" "cli" {
  for_each = data.databricks_clusters.all.ids
  cluster_id = each.key
  pypi {
    package = "databricks-cli"
  }
}
```

Fix #904 #954
nfx added a commit that referenced this issue Dec 15, 2021
```hcl
data "databricks_clusters" "all" {
}

resource "databricks_library" "cli" {
  for_each = data.databricks_clusters.all.ids
  cluster_id = each.key
  pypi {
    package = "databricks-cli"
  }
}
```

Fix #904 #954
@nfx nfx closed this as completed in #971 Dec 15, 2021
nfx added a commit that referenced this issue Dec 16, 2021
* Added `databricks_library` resource to install library on `databricks_cluster` ([#904](#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([#955](#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([#952](#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([#599](#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([#957](#957)).
* Fixed disabling `databricks_global_init_script` ([#958](#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([#981](#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([#965](#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([#966](#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([#964](#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
@nfx nfx mentioned this issue Dec 16, 2021
nfx added a commit that referenced this issue Dec 16, 2021
Change log:

* Added `databricks_library` resource to install library on `databricks_cluster` ([#904](#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([#955](#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([#952](#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([#599](#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([#957](#957)).
* Fixed disabling `databricks_global_init_script` ([#958](#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([#981](#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([#965](#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([#966](#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([#964](#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
michael-berk pushed a commit to michael-berk/terraform-provider-databricks that referenced this issue Feb 15, 2023
Change log:

* Added `databricks_library` resource to install library on `databricks_cluster` ([databricks#904](databricks#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([databricks#955](databricks#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([databricks#952](databricks#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([databricks#599](databricks#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([databricks#957](databricks#957)).
* Fixed disabling `databricks_global_init_script` ([databricks#958](databricks#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([databricks#981](databricks#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([databricks#965](databricks#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([databricks#966](databricks#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([databricks#964](databricks#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant