Added databricks bundle generate job command
#1043
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewnester
changed the title
databricks bundle generate job command
cmd/bundle/generate/job.go
Outdated
| } | ||
|
|
||
| for _, task := range job.Settings.Tasks { | ||
| err := downloadNotebookAndReplaceTaskPath(ctx, &task, w, outputDir) |
There was a problem hiding this comment.
We should also do this for SparkPythonTask. Are we also planning to do this for wheel and jar tasks?
There was a problem hiding this comment.
@shreyas-goenka eventually yes, for now it's only scoped to notebook tasks
There was a problem hiding this comment.
Note that we can't really fully support wheel and jar tasks without user intervention, since we'd want the Python/Scala sources, not the wheel or jar file. The job doesn't know about those sources.
pietern
reviewed
Dec 7, 2023
libs/config/value.go
Outdated
| // See convert.convertMap for details | ||
| sort.SliceStable(keys, func(i, j int) bool { | ||
| return m[keys[i]].Location().Line < m[keys[j]].Location().Line | ||
| }) |
andrewnester
requested review from
pietern,
lennartkats-db and
shreyas-goenka
nfx
previously requested changes
Dec 13, 2023
andrewnester
force-pushed
the
generate-yaml-notebook
branch
from
e7b5ec9 to
882c8bc
Compare
pietern
reviewed
Dec 18, 2023
andrewnester
force-pushed
the
generate-yaml-notebook
branch
from
ee458ca to
316eda4
Compare
andrewnester
force-pushed
the
generate-yaml-notebook
branch
from
f13e585 to
2777a29
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1043 +/- ##
==========================================
+ Coverage 49.24% 49.50% +0.26%
==========================================
Files 273 281 +8
Lines 10570 10711 +141
==========================================
+ Hits 5205 5303 +98
- Misses 4810 4844 +34
- Partials 555 564 +9 ☔ View full report in Codecov by Sentry. |
pietern
reviewed
Jan 16, 2024
pietern
reviewed
Jan 17, 2024
pietern
approved these changes
Jan 17, 2024
pietern
reviewed
Jan 17, 2024
andrewnester
added a commit
that referenced
this pull request
Jan 17, 2024
CLI: * Fix windows style file paths in fs cp command ([#1118](#1118)). * Do not require positional arguments if they should be provided in JSON ([#1125](#1125)). * Always require path parameters as positional arguments ([#1129](#1129)). Bundles: * Add debug log line for when bundle init is run from non-TTY interface ([#1117](#1117)). * Added `databricks bundle generate job` command ([#1043](#1043)). * Support passing job parameters to bundle run ([#1115](#1115)). Dependency updates: * Bump golang.org/x/oauth2 from 0.15.0 to 0.16.0 ([#1124](#1124)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 17, 2024
CLI: * Fix windows style file paths in fs cp command ([#1118](#1118)). * Do not require positional arguments if they should be provided in JSON ([#1125](#1125)). * Always require path parameters as positional arguments ([#1129](#1129)). Bundles: * Add debug log line for when bundle init is run from non-TTY interface ([#1117](#1117)). * Added `databricks bundle generate job` command ([#1043](#1043)). * Support passing job parameters to bundle run ([#1115](#1115)). Dependency updates: * Bump golang.org/x/oauth2 from 0.15.0 to 0.16.0 ([#1124](#1124)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Now it's possible to generate bundle configuration for existing job.
For now it only supports jobs with notebook tasks.
It will download notebooks referenced in the job tasks and generate bundle YAML config for this job which can be included in larger bundle.
Tests
Running command manually
Example of generated config
Tests
Manual (on our last 100 jobs) + added end-to-end test