Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable locking for development mode #1302

Merged
merged 9 commits into from
Apr 18, 2024
Merged

Disable locking for development mode #1302

merged 9 commits into from
Apr 18, 2024

Conversation

lennartkats-db
Copy link
Contributor

@lennartkats-db lennartkats-db commented Mar 21, 2024 •
edited
Loading

Changes

This changes databricks bundle deploy so that it skips the lock acquisition/release step for a mode: development target:

  • This saves about 2 seconds (measured over 100 runs on a quiet/busy workspace).
  • This helps avoid the deploy lock acquired by lennart@company.com at 2024-02-28 15:48:38.40603 +0100 CET. Use --force-lock to override error
  • Risk: this may cause deployment conflicts, but since dev mode deployments are always scoped to a user, that risk should be minimal

Update after discussion:

Measurements

100 deployments of the "python_default" project to an empty workspace

Before this branch:
p50 time: 11.479 seconds
p90 time: 11.757 seconds

After this branch:
p50 time: 9.386 seconds
p90 time: 9.599 seconds

100 deployments of the "python_default" project to a busy (staging) workspace

Before this branch:

  • p50 time: 13.335 seconds
  • p90 time: 15.295 seconds

After this branch:

  • p50 time: 11.397 seconds
  • p90 time: 11.743 seconds

Typical duration of deployment steps

  • Acquiring Deployment Lock: 1.096 seconds
  • Deployment Preparations and Operations: 1.477 seconds
  • Uploading Artifacts: 1.26 seconds
  • Finalizing Deployment: 9.699 seconds
  • Releasing Deployment Lock: 1.198 seconds

@codecov-commenter
Copy link

codecov-commenter commented Mar 21, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 64.70588% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 54.10%. Comparing base (e22dd8a) to head (e8937ef).
Report is 59 commits behind head on main.

Files Patch % Lines
bundle/config/lock.go 0.00% 4 Missing ⚠️
bundle/config/mutator/process_target_mode.go 84.61% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1302      +/-   ##
==========================================
+ Coverage   52.25%   54.10%   +1.85%     
==========================================
  Files         317      328      +11     
  Lines       18004    15076    -2928     
==========================================
- Hits         9408     8157    -1251     
+ Misses       7903     6159    -1744     
- Partials      693      760      +67

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lennartkats-db lennartkats-db changed the title [RFC] Disable locking for development mode Disable locking for development mode Mar 29, 2024
@lennartkats-db lennartkats-db force-pushed the faster-dev-mode-kill-kill branch from d37b54e to 032981f Compare March 29, 2024 13:42
@lennartkats-db lennartkats-db enabled auto-merge April 3, 2024 18:50
andrewnester
andrewnester approved these changes Apr 9, 2024
View reviewed changes
bundle/config/lock.go Outdated Show resolved Hide resolved
@lennartkats-db lennartkats-db added this pull request to the merge queue Apr 9, 2024
@andrewnester andrewnester removed this pull request from the merge queue due to a manual request Apr 9, 2024
@pietern @andrewnester
Update bundle/config/lock.go
a68c95c 
Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com>
@lennartkats-db lennartkats-db added this pull request to the merge queue Apr 18, 2024
Merged via the queue into databricks:main with commit c3a7d17 Apr 18, 2024
4 of 5 checks passed
@lennartkats-db lennartkats-db deleted the faster-dev-mode-kill-kill branch April 18, 2024 02:06
pietern added a commit that referenced this pull request Apr 23, 2024
@pietern
Release v0.218.0
29425f5 
This release marks the general availability of Databricks Asset Bundles.

CLI:
 * Publish Docker images ([#1353](#1353)).
 * Add support for multi-arch Docker images ([#1362](#1362)).
 * Do not prefill https:// in prompt for Databricks Host ([#1364](#1364)).
 * Add better documentation for the `auth login` command ([#1366](#1366)).
 * Add URLs for authentication documentation to the auth command help ([#1365](#1365)).

Bundles:
 * Fix compute override for foreach tasks ([#1357](#1357)).
 * Transform artifact files source patterns in build not upload stage ([#1359](#1359)).
 * Convert between integer and float in normalization ([#1371](#1371)).
 * Disable locking for development mode ([#1302](#1302)).
 * Resolve variable references inside variable lookup fields ([#1368](#1368)).
 * Added validate mutator to surface additional bundle warnings ([#1352](#1352)).
 * Upgrade terraform-provider-databricks to 1.40.0 ([#1376](#1376)).
 * Print host in `bundle validate` when passed via profile or environment variables ([#1378](#1378)).
 * Cleanup remote file path on bundle destroy ([#1374](#1374)).
 * Add docs URL for `run_as` in error message ([#1381](#1381)).
 * Enable job queueing by default ([#1385](#1385)).
 * Added support for job environments ([#1379](#1379)).
 * Processing and completion of positional args to bundle run ([#1120](#1120)).
 * Add legacy option for `run_as` ([#1384](#1384)).

API Changes:
 * Changed `databricks lakehouse-monitors cancel-refresh` command with new required argument order.
 * Changed `databricks lakehouse-monitors create` command with new required argument order.
 * Changed `databricks lakehouse-monitors delete` command with new required argument order.
 * Changed `databricks lakehouse-monitors get` command with new required argument order.
 * Changed `databricks lakehouse-monitors get-refresh` command with new required argument order.
 * Changed `databricks lakehouse-monitors list-refreshes` command with new required argument order.
 * Changed `databricks lakehouse-monitors run-refresh` command with new required argument order.
 * Changed `databricks lakehouse-monitors update` command with new required argument order.
 * Changed `databricks account workspace-assignment update` command to return response.

OpenAPI commit 94684175b8bd65f8701f89729351f8069e8309c9 (2024-04-11)

Dependency updates:
 * Bump github.com/databricks/databricks-sdk-go from 0.37.0 to 0.38.0 ([#1361](#1361)).
 * Bump golang.org/x/net from 0.22.0 to 0.23.0 ([#1380](#1380)).
@pietern pietern mentioned this pull request Apr 23, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Apr 23, 2024
@pietern
Release v0.218.0 (#1391)
27d35d5 
This release marks the general availability of Databricks Asset Bundles.

CLI:
* Publish Docker images
([#1353](#1353)).
* Add support for multi-arch Docker images
([#1362](#1362)).
* Do not prefill https:// in prompt for Databricks Host
([#1364](#1364)).
* Add better documentation for the `auth login` command
([#1366](#1366)).
* Add URLs for authentication documentation to the auth command help
([#1365](#1365)).

Bundles:
* Fix compute override for foreach tasks
([#1357](#1357)).
* Transform artifact files source patterns in build not upload stage
([#1359](#1359)).
* Convert between integer and float in normalization
([#1371](#1371)).
* Disable locking for development mode
([#1302](#1302)).
* Resolve variable references inside variable lookup fields
([#1368](#1368)).
* Added validate mutator to surface additional bundle warnings
([#1352](#1352)).
* Upgrade terraform-provider-databricks to 1.40.0
([#1376](#1376)).
* Print host in `bundle validate` when passed via profile or environment
variables ([#1378](#1378)).
* Cleanup remote file path on bundle destroy
([#1374](#1374)).
* Add docs URL for `run_as` in error message
([#1381](#1381)).
* Enable job queueing by default
([#1385](#1385)).
* Added support for job environments
([#1379](#1379)).
* Processing and completion of positional args to bundle run
([#1120](#1120)).
* Add legacy option for `run_as`
([#1384](#1384)).

API Changes:
* Changed `databricks lakehouse-monitors cancel-refresh` command with
new required argument order.
* Changed `databricks lakehouse-monitors create` command with new
required argument order.
* Changed `databricks lakehouse-monitors delete` command with new
required argument order.
* Changed `databricks lakehouse-monitors get` command with new required
argument order.
* Changed `databricks lakehouse-monitors get-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors list-refreshes` command with
new required argument order.
* Changed `databricks lakehouse-monitors run-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors update` command with new
required argument order.
* Changed `databricks account workspace-assignment update` command to
return response.

OpenAPI commit 94684175b8bd65f8701f89729351f8069e8309c9 (2024-04-11)

Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.37.0 to 0.38.0
([#1361](#1361)).
* Bump golang.org/x/net from 0.22.0 to 0.23.0
([#1380](#1380)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietern pietern pietern approved these changes

@andrewnester andrewnester andrewnester approved these changes

@fjakobs fjakobs Awaiting requested review from fjakobs

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lennartkats-db @codecov-commenter @pietern @andrewnester