Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make databricks configure save only explicit fields #973

Merged
merged 4 commits into from
Nov 10, 2023
Merged

Make databricks configure save only explicit fields #973

merged 4 commits into from
Nov 10, 2023

Conversation

ilia-db
Copy link
Contributor

@ilia-db ilia-db commented Nov 9, 2023

Changes

Save only explicit fields to the config file
This applies to two commands: configure and auth login.
The latter only pulls env vars in the case of the --configure-cluster flag

Tests

Manual, plus additional unit test for the configure command

@ilia-db ilia-db requested review from pietern and mgyucht November 9, 2023 15:47
mgyucht
mgyucht approved these changes Nov 9, 2023
View reviewed changes
Copy link
Contributor

@mgyucht mgyucht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thank you for testing this!

@mgyucht mgyucht enabled auto-merge November 9, 2023 15:51
@mgyucht mgyucht disabled auto-merge November 9, 2023 15:51
pietern
pietern reviewed Nov 9, 2023
View reviewed changes
cmd/configure/configure_test.go

t.Setenv("DATABRICKS_HOST", "https://host")
t.Setenv("DATABRICKS_AUTH_TYPE", "metadata-service")
t.Setenv("DATABRICKS_METADATA_SERVICE_URL", "https://metadata")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also set the DATABRICKS_TOKEN env var to some value to make sure it doesn't have precedence over the value that the user provides through stdin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current logic it will pick up the token from the env var and won't even try to ask for a new one from a user (or read from sdtin). From git blame I figured it is a feature, as it was added explicitly - the CLI is able to bootstrap the config based on the environment variables, and if there's enough of them, it will pretty much create a config automatically without any additional prompts.

To me it feels a bit counter intuitive, I would expect databricks configure to always ask all of its inputs. With the current logic it should be named databrikcs write-to-config instead, not configure (since in some cases you don't even configure anything).

Would it make sense to remove reading from the environment completely?

cli/cmd/configure/configure.go

Line 140 in 3284a8c

err := config.ConfigAttributes.Configure(&cfg)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And a side note: databricks vscode extension doesn't set the token env variable, only host with metadata

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be users who set these env vars and expect it to work (predominantly for automation), so let's keep it in tact. I have seen cases where the token is passed through stdin, the rest being set through env vars. Looking at the test context I figured the other env vars would also be masked out, but since it makes sense, disregard my comment.

Do we already have a test confirming that this behavior works? I.e. pass the host and token as env vars and the command silently succeeds and writes out a profile?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a test to confirm that configure works with both token and host taken from the env vars

cmd/configure/configure_test.go Outdated Show resolved Hide resolved
@ilia-db ilia-db requested a review from pietern November 10, 2023 09:58
pietern
pietern approved these changes Nov 10, 2023
View reviewed changes
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking more about the pattern of specifying env vars to pick them up in the profile: given that this command requires the token field to be present, I suspect it is only used for PAT-based auth, and if so, then we don't need to worry about the other configuration fields. If it supported more auth types, we would need to be more careful.

@pietern pietern changed the title Save only explicit fields to the config file Make databricks configure save only explicit fields Nov 10, 2023
@ilia-db ilia-db added this pull request to the merge queue Nov 10, 2023
Merged via the queue into main with commit e82a49b Nov 10, 2023
@ilia-db ilia-db deleted the avoid-env-vars branch November 10, 2023 14:11
pietern added a commit that referenced this pull request Nov 29, 2023
@pietern
Release v0.210.0
e710c52 
This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects.

CLI:
 * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)).
 * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)).
 * Make `databricks configure` save only explicit fields ([#973](#973)).
 * Add `databricks labs` command group ([#914](#914)).
 * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)).
 * Add `--configure-cluster` flag to configure command ([#1005](#1005)).
 * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)).

Bundles:
 * Do not allow empty descriptions for bundle template inputs ([#967](#967)).
 * Added support for top-level permissions ([#928](#928)).
 * Allow jobs to be manually unpaused in development mode ([#885](#885)).
 * Fix template initialization from current working directory ([#976](#976)).
 * Add `--tag` and `--branch` options to bundle init command ([#975](#975)).
 * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)).
 * Enable `spark_jar_task` with local JAR libraries ([#993](#993)).
 * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)).
 * Improve error message when path is not a bundle template ([#985](#985)).
 * Correctly overwrite local state if remote state is newer ([#1008](#1008)).
 * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)).
 * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)).
 * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)).

Internal:
 * Update CLI version in the VS Code extension during release ([#1014](#1014)).

API Changes:
 * Changed `databricks functions create` command . New request type is .
 * Changed `databricks metastores create` command with new required argument order.
 * Removed `databricks metastores enable-optimization` command.
 * Removed `databricks account o-auth-enrollment` command group.
 * Removed `databricks apps delete` command.
 * Removed `databricks apps get` command.
 * Added `databricks apps delete-app` command.
 * Added `databricks apps get-app` command.
 * Added `databricks apps get-app-deployment-status` command.
 * Added `databricks apps get-apps` command.
 * Added `databricks apps get-events` command.
 * Added `databricks account network-connectivity` command group.

OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23)

Dependency updates:
 * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)).
 * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)).
 * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)).
 * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)).
 * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).
@pietern pietern mentioned this pull request Nov 29, 2023
Merged
github-merge-queue bot pushed a commit that referenced this pull request Nov 29, 2023
@pietern
Release v0.210.0 (#1023)
09991da 
This release includes the new `databricks labs` command to install,
manage, and run Databricks Labs projects.

CLI:
* Add `--debug` as shortcut for `--log-level debug`
([#964](#964)).
* Improved usability of `databricks auth login ... --configure-cluster`
([#956](#956)).
* Make `databricks configure` save only explicit fields
([#973](#973)).
* Add `databricks labs` command group
([#914](#914)).
* Tolerate missing .databrickscfg file during `databricks auth login`
([#1003](#1003)).
* Add `--configure-cluster` flag to configure command
([#1005](#1005)).
* Fix bug where the account or workspace client could be `nil`
([#1020](#1020)).

Bundles:
* Do not allow empty descriptions for bundle template inputs
([#967](#967)).
* Added support for top-level permissions
([#928](#928)).
* Allow jobs to be manually unpaused in development mode
([#885](#885)).
* Fix template initialization from current working directory
([#976](#976)).
* Add `--tag` and `--branch` options to bundle init command
([#975](#975)).
* Work around DLT issue with `$PYTHONPATH` not being set correctly
([#999](#999)).
* Enable `spark_jar_task` with local JAR libraries
([#993](#993)).
* Pass `USERPROFILE` environment variable to Terraform
([#1001](#1001)).
* Improve error message when path is not a bundle template
([#985](#985)).
* Correctly overwrite local state if remote state is newer
([#1008](#1008)).
* Add mlops-stacks to the default `databricks bundle init` prompt
([#988](#988)).
* Do not add wheel content hash in uploaded Python wheel path
([#1015](#1015)).
* Do not replace pipeline libraries if there are no matches for pattern
([#1021](#1021)).

Internal:
* Update CLI version in the VS Code extension during release
([#1014](#1014)).

API Changes:
 * Changed `databricks functions create` command.
* Changed `databricks metastores create` command with new required
argument order.
 * Removed `databricks metastores enable-optimization` command.
 * Removed `databricks account o-auth-enrollment` command group.
 * Removed `databricks apps delete` command.
 * Removed `databricks apps get` command.
 * Added `databricks apps delete-app` command.
 * Added `databricks apps get-app` command.
 * Added `databricks apps get-app-deployment-status` command.
 * Added `databricks apps get-apps` command.
 * Added `databricks apps get-events` command.
 * Added `databricks account network-connectivity` command group.

OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23)

Dependency updates:
* Bump golang.org/x/term from 0.13.0 to 0.14.0
([#981](#981)).
* Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0
([#979](#979)).
* Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0
([#982](#982)).
* Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0
([#980](#980)).
* Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0
([#1019](#1019)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mgyucht mgyucht mgyucht approved these changes

@pietern pietern pietern approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ilia-db @pietern @mgyucht