Added test to submit and run various Python tasks on multiple DBR versions #806

andrewnester · 2023-09-26T14:04:08Z

Changes

These tests allow us to get information for execution context (PYTHONPATH, CWD) for various Python tasks and different cluster setups.

Note: this test won't be executed automatically as part of nightly builds since it requires RUN_PYTHON_TASKS_TEST env to be executed.

Tests

Integration test run successfully.

…sions

nfx · 2023-09-26T18:39:45Z

internal/bundle/helpers.go

+var sparkVersions = []string{
+	"11.3.x-scala2.12",
+	"12.2.x-scala2.12",
+	"13.0.x-scala2.12",
+	"13.1.x-scala2.12",
+	"13.2.x-scala2.12",
+	"13.3.x-scala2.12",
+	"14.0.x-scala2.12",
+	"14.1.x-scala2.12",
+}


this list will get out of date in couple of months, can you use the w.Clusters.SparkVersions to pick the list of runtimes dynamically?
see https://github.com/databricks/databricks-sdk-py/blob/main/tests/integration/test_auth.py#L90-L124

I consider that but for now we want to keep it simple since we don't run them continuously yet. Once we do, we will get the list dynamically indeed

nfx · 2023-09-26T18:42:23Z

internal/bundle/helpers.go

+				SparkVersion:     sparkVersions[i],
+				NumWorkers:       1,
+				NodeTypeId:       nodeTypeId,
+				DataSecurityMode: compute.DataSecurityModeUserIsolation,


this is only for Unity Catalog-enabled workspaces, but it doesn't check for .DataSecurityModeSingleUser and DataSecurityModeNone.

what about non-uc workspaces, like DataSecurityModeLegacySingleUser?

internal/bundle/helpers.go

nfx · 2023-09-26T18:45:37Z

internal/bundle/python_tasks_test.go

+
+func TestAccRunPythonTaskWorkspace(t *testing.T) {
+	// TODO: remove RUN_PYTHON_TASKS_TEST when ready to be executed as part of nightly
+	internal.GetEnvOrSkipTest(t, "RUN_PYTHON_TASKS_TEST")


there's ucws testing environment, which tests against UC-enabled workspace, use those env vars

atm we don't plan to execute it on continuously basis yet, just as a script to be executed manually to generate the data

yep, but you can still execute it with deco env shell and pick a *-ucws environment

pietern

General comment; since we don't run this as part of the regular suites, I think it should live in its own package. The helpers you introduce here are generic enough that they should be reusable from more testing packages.

internal/bundle/helpers.go

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

pietern · 2023-10-02T09:04:39Z

internal/python/testdata/my_test_code-0.0.1-py3-none-any.whl

Please include the source for this wheel file.

pietern

Stamp to unblock.

unblock merge

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

…sions (#806) ## Changes These tests allow us to get information for execution context (PYTHONPATH, CWD) for various Python tasks and different cluster setups. Note: this test won't be executed automatically as part of nightly builds since it requires RUN_PYTHON_TASKS_TEST env to be executed. ## Tests Integration test run successfully. --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

Added test to submit and run various Python tasks on multiple DBR ver…

353e75a

…sions

andrewnester requested review from pietern and shreyas-goenka September 26, 2023 14:04

correctly skip files

e5b6dbd

nfx previously requested changes Sep 26, 2023

View reviewed changes

pietern reviewed Sep 27, 2023

View reviewed changes

internal/bundle/helpers.go Outdated Show resolved Hide resolved

internal/bundle/helpers.go Outdated Show resolved Hide resolved

internal/bundle/helpers.go Outdated Show resolved Hide resolved

andrewnester and others added 3 commits September 27, 2023 09:50

Update internal/bundle/helpers.go

0dfdd3e

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

Merge branch 'main' into run-python-task-everywhere

514b5c9

refactored

828a463

andrewnester requested a review from pietern September 27, 2023 10:09

use CLI repo as repo basis

aa5d241

pietern reviewed Oct 2, 2023

View reviewed changes

internal/python/testdata/my_test_code-0.0.1-py3-none-any.whl Outdated

Copy link

Contributor

pietern Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include the source for this wheel file.

pietern approved these changes Oct 2, 2023

View reviewed changes

andrewnester enabled auto-merge October 3, 2023 11:11

Merge branch 'main' into run-python-task-everywhere

720b617

andrewnester requested a review from nfx October 3, 2023 11:11

andrewnester disabled auto-merge October 3, 2023 11:17

andrewnester added this pull request to the merge queue Oct 3, 2023

Merged via the queue into main with commit 79e271f Oct 3, 2023

andrewnester deleted the run-python-task-everywhere branch October 3, 2023 11:23

andrewnester mentioned this pull request Oct 4, 2023

Release v0.207.0 #839

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added test to submit and run various Python tasks on multiple DBR versions #806

Added test to submit and run various Python tasks on multiple DBR versions #806

andrewnester commented Sep 26, 2023

nfx Sep 26, 2023

andrewnester Sep 27, 2023

nfx Sep 26, 2023

nfx Sep 26, 2023

andrewnester Sep 27, 2023

nfx Sep 27, 2023

pietern left a comment

pietern Oct 2, 2023

pietern left a comment

Added test to submit and run various Python tasks on multiple DBR versions #806

Added test to submit and run various Python tasks on multiple DBR versions #806

Conversation

andrewnester commented Sep 26, 2023

Changes

Tests

nfx Sep 26, 2023

Choose a reason for hiding this comment

andrewnester Sep 27, 2023

Choose a reason for hiding this comment

nfx Sep 26, 2023

Choose a reason for hiding this comment

nfx Sep 26, 2023

Choose a reason for hiding this comment

andrewnester Sep 27, 2023

Choose a reason for hiding this comment

nfx Sep 27, 2023

Choose a reason for hiding this comment

pietern left a comment

Choose a reason for hiding this comment

pietern Oct 2, 2023

Choose a reason for hiding this comment

pietern left a comment

Choose a reason for hiding this comment