Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PythonMutator: explain missing package error #1736

Merged
merged 2 commits into from
Sep 2, 2024

Conversation

kanterov
Copy link
Contributor

Changes

Explain the error when the databricks-pydabs package is not installed or the Python environment isn't correctly activated.

Example output:

Error: python mutator process failed: ".venv/bin/python3 -m databricks.bundles.build --phase load --input .../input.json --output .../output.json --diagnostics .../diagnostics.json: exit status 1", use --debug to enable logging

.../.venv/bin/python3: Error while finding module specification for 'databricks.bundles.build' (ModuleNotFoundError: No module named 'databricks')

Explanation: 'databricks-pydabs' library is not installed in the Python environment.

If using Python wheels, ensure that 'databricks-pydabs' is included in the dependencies, 
and that the wheel is installed in the Python environment:

  $ .venv/bin/pip install -e .

If using a virtual environment, ensure it is specified as the venv_path property in databricks.yml, 
or activate the environment before running CLI commands:

  experimental:
    pydabs:
      venv_path: .venv

Tests

Unit tests

@kanterov kanterov force-pushed the kanterov/pydabs-explain-errors branch from 7c11433 to b9ae868 Compare August 30, 2024 11:30
`

func explainProcessErr(stderr string) string {
if strings.Contains(stderr, "Error while finding module specification for 'databricks.bundles.build'") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this error message portable across Python versions and OSes?

If so, please include a comment confirming this for future readers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked different implementations for Python. It's very reliable, there are even unit tests using similar regex. The code that produces this message is written in Python and is shared by implementations like pypy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking!

go.mod Outdated
@@ -1,6 +1,6 @@
module github.com/databricks/cli

go 1.22
go 1.22.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will. But it breaks my dev environment because it can't find toolchain for 1.22. Do you have this problem? I use IntelliJ

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not with VS Code (nor command line). Does it maybe use a bundled Go toolchain?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find where toolchain is configured, but thanks for pointers, I will try to figure it out 👍

@kanterov kanterov force-pushed the kanterov/pydabs-explain-errors branch from b9ae868 to 14003a2 Compare August 30, 2024 11:51
`

func explainProcessErr(stderr string) string {
if strings.Contains(stderr, "Error while finding module specification for 'databricks.bundles.build'") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking!

@pietern pietern added this pull request to the merge queue Aug 30, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Aug 30, 2024
@pietern
Copy link
Contributor

pietern commented Sep 2, 2024

@kanterov This is blocked on a merge conflict.

auto-merge was automatically disabled September 2, 2024 07:32

Head branch was pushed to by a user without write access

@kanterov kanterov force-pushed the kanterov/pydabs-explain-errors branch from bb7612f to 4fe4b7e Compare September 2, 2024 07:32
@kanterov kanterov marked this pull request as draft September 2, 2024 09:10
@kanterov kanterov force-pushed the kanterov/pydabs-explain-errors branch from 2290801 to 9f70916 Compare September 2, 2024 09:11
@kanterov kanterov force-pushed the kanterov/pydabs-explain-errors branch from 9f70916 to 39b4e16 Compare September 2, 2024 09:15
@kanterov kanterov marked this pull request as ready for review September 2, 2024 09:30
@kanterov kanterov requested a review from pietern September 2, 2024 09:40
@kanterov
Copy link
Contributor Author

kanterov commented Sep 2, 2024

@pietern fixed, can you please add to the merge queue?

@pietern pietern added this pull request to the merge queue Sep 2, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 2, 2024
@pietern pietern added this pull request to the merge queue Sep 2, 2024
Merged via the queue into databricks:main with commit ed44881 Sep 2, 2024
5 checks passed
andrewnester added a commit that referenced this pull request Sep 5, 2024
CLI:
 * Do not error if we cannot prompt for a profile in `auth login` ([#1745](#1745)).

Bundles:
 * Pass along $AZURE_CONFIG_FILE to Terraform process ([#1734](#1734)).
 * Add prompt when a pipeline recreation happens ([#1672](#1672)).
 * Use materialized views in the default-sql template ([#1709](#1709)).
 * Update templates to latest LTS DBR ([#1715](#1715)).
 * Make lock optional in the JSON schema ([#1738](#1738)).
 * Do not suppress normalisation diagnostics for resolving variables ([#1740](#1740)).
 * Include a permissions section in all templates ([#1713](#1713)).
 * Fixed complex variables are not being correctly merged from include files ([#1746](#1746)).
 * Fixed variable override in target with full variable syntax ([#1749](#1749)).

Internal:
 * Consider serverless clusters as compatible for Python wheel tasks ([#1733](#1733)).
 * PythonMutator: explain missing package error ([#1736](#1736)).
 * Add `dyn.Time` to box a timestamp with its original string value ([#1732](#1732)).
 * Fix streaming of stdout, stdin, stderr in cobra test runner ([#1742](#1742)).

Dependency updates:
 * Bump github.com/Masterminds/semver/v3 from 3.2.1 to 3.3.0 ([#1741](#1741)).
github-merge-queue bot pushed a commit that referenced this pull request Sep 5, 2024
CLI:
* Do not error if we cannot prompt for a profile in `auth login`
([#1745](#1745)).

Bundles:

As of this release CLI will show a prompt is if there are configuration
changes which will lead to a DLT recreation.
Users can skip the prompt by specifying the `--auto-approve` flag

* Pass along $AZURE_CONFIG_FILE to Terraform process
([#1734](#1734)).
* Add prompt when a pipeline recreation happens
([#1672](#1672)).
* Use materialized views in the default-sql template
([#1709](#1709)).
* Update templates to latest LTS DBR
([#1715](#1715)).
* Make lock optional in the JSON schema
([#1738](#1738)).
* Do not suppress normalisation diagnostics for resolving variables
([#1740](#1740)).
* Include a permissions section in all templates
([#1713](#1713)).
* Fixed complex variables are not being correctly merged from include
files ([#1746](#1746)).
* Fixed variable override in target with full variable syntax
([#1749](#1749)).

Internal:
* Consider serverless clusters as compatible for Python wheel tasks
([#1733](#1733)).
* PythonMutator: explain missing package error
([#1736](#1736)).
* Add `dyn.Time` to box a timestamp with its original string value
([#1732](#1732)).
* Fix streaming of stdout, stdin, stderr in cobra test runner
([#1742](#1742)).

Dependency updates:
* Bump github.com/Masterminds/semver/v3 from 3.2.1 to 3.3.0
([#1741](#1741)).

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants