Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an experimental default-sql template #1051

Merged
merged 35 commits into from
Feb 19, 2024
Merged

Add an experimental default-sql template #1051

merged 35 commits into from
Feb 19, 2024

Conversation

lennartkats-db
Copy link
Contributor

@lennartkats-db lennartkats-db commented Dec 11, 2023 •
edited
Loading

Changes

This adds a default-sql template!

In this latest revision, I've hidden the new template from the list so we can merge it, iterate over it, and properly release the template at the right time.

  • WorkspaceFS support for .sql files is in prod
  • SQL extension is preconfigured based on extension settings (if possible)
  • Streaming tables support is either ungated or the template provides instructions about signup
    • Mitigation for now: this template is hidden from the list of templates.
  • Change template to use USE CATALOG when supported
  • Support non-UC workspaces

Tests

  • Unit tests
  • Manual testing
  • More manual testing
  • Reviewer testing

@lennartkats-db lennartkats-db mentioned this pull request Dec 11, 2023
Merged
@lennartkats-db lennartkats-db changed the title [DRAFT] default-sql template [DRAFT] Add a default-sql template Dec 11, 2023
@lennartkats-db lennartkats-db changed the title [DRAFT] Add a default-sql template Add a default-sql template Dec 11, 2023
PaulCornellDB
PaulCornellDB reviewed Dec 13, 2023
View reviewed changes
cmd/bundle/init.go Outdated Show resolved Hide resolved
@lennartkats-db lennartkats-db changed the title Add a default-sql template [DONTMERGE] Add a default-sql template Dec 18, 2023
@pietern
Copy link
Contributor

pietern commented Feb 5, 2024

@lennartkats-db The same holds here as in #1059 -- the warehouse patterns looks like it may not work.

@lennartkats-db lennartkats-db removed the request for review from fjakobs February 19, 2024 09:12
@lennartkats-db lennartkats-db requested review from fjakobs and removed request for fjakobs February 19, 2024 11:11
fjakobs
fjakobs requested changes Feb 19, 2024
View reviewed changes
libs/template/templates/default-sql/library/versions.tmpl Show resolved Hide resolved
libs/template/templates/default-sql/template/{{.project_name}}/.vscode/settings.json.tmpl Show resolved Hide resolved
libs/template/templates/default-sql/template/{{.project_name}}/scratch/README.md Outdated
@@ -0,0 +1,4 @@
# scratch

This folder is reserved for personal, exploratory notebooks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

notebooks and SQL files.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just talk about SQL files here. If we wanted to support notebooks then we should be adding a requirements.txt to configure the python env.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added SQL files. And notebooks are actually very useful for SQL at this point. They can be used with warehouses too (though probably not in the IDE, yet?)

Re. requirements.txt: is that used for notebooks?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notebooks in the IDE currently use spark.sql so it's technically not the same. I could imagine adding a SQL mode to the notebooks just like in the webapp but that's currently not scoped.

Notebooks can be used without virtual envs but that would clutter the global Python module space. If we recommend notebooks then using virtual envs and having a requirements.txt would be best practice.

@fjakobs fjakobs self-requested a review February 19, 2024 12:00
@lennartkats-db lennartkats-db added this pull request to the merge queue Feb 19, 2024
Merged via the queue into databricks:main with commit 162b115 Feb 19, 2024
4 checks passed
@lennartkats-db lennartkats-db deleted the sql-template branch February 19, 2024 12:07
@lennartkats-db lennartkats-db mentioned this pull request Feb 20, 2024
Merged
andrewnester added a commit that referenced this pull request Feb 20, 2024
@andrewnester
Release v0.214.0
7ccede1 
CLI:
 * Add support for UC Volumes to the `databricks fs` commands ([#1209](#1209)).

Bundles:
 * Use dynamic configuration model in bundles ([#1098](#1098)).
 * Allow use of variables references in primitive non-string fields ([#1219](#1219)).
 * Add an experimental default-sql template ([#1051](#1051)).
 * Add an experimental dbt-sql template ([#1059](#1059)).

Internal:
 * Add fork-user to winget release workflow ([#1214](#1214)).
 * Use `any` as type for data sources and resources in `tf/schema` ([#1216](#1216)).
 * Avoid infinite recursion when normalizing a recursive type ([#1213](#1213)).
 * Fix issue where interpolating a new ref would rewrite unrelated fields ([#1217](#1217)).
 * Use `dyn.Value` as input to generating Terraform JSON ([#1218](#1218)).

API Changes:
 * Changed `databricks lakehouse-monitors update` command with new required argument order.
 * Added `databricks online-tables` command group.

OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19)
Dependency updates:
 * Bump Terraform provider to v1.36.2 ([#1215](#1215)).
 * Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0 ([#1222](#1222)).
@andrewnester andrewnester mentioned this pull request Feb 20, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Feb 20, 2024
@andrewnester
Release v0.214.0 (#1226)
2683341 
CLI:
* Add support for UC Volumes to the `databricks fs` commands
([#1209](#1209)).

Bundles:
* Use dynamic configuration model in bundles
([#1098](#1098)).
* Allow use of variables references in primitive non-string fields
([#1219](#1219)).
* Add an experimental default-sql template
([#1051](#1051)).
* Add an experimental dbt-sql template
([#1059](#1059)).

Internal:
* Add fork-user to winget release workflow
([#1214](#1214)).
* Use `any` as type for data sources and resources in `tf/schema`
([#1216](#1216)).
* Avoid infinite recursion when normalizing a recursive type
([#1213](#1213)).
* Fix issue where interpolating a new ref would rewrite unrelated fields
([#1217](#1217)).
* Use `dyn.Value` as input to generating Terraform JSON
([#1218](#1218)).

API Changes:
* Changed `databricks lakehouse-monitors update` command with new
required argument order.
 * Added `databricks online-tables` command group.

OpenAPI commit cdd76a98a4fca7008572b3a94427566dd286c63b (2024-02-19)
Dependency updates:
* Bump Terraform provider to v1.36.2
([#1215](#1215)).
* Bump github.com/databricks/databricks-sdk-go from 0.32.0 to 0.33.0
([#1222](#1222)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juliacrawf-db juliacrawf-db juliacrawf-db left review comments

@PaulCornellDB PaulCornellDB PaulCornellDB left review comments

@fjakobs fjakobs fjakobs approved these changes

@pietern pietern pietern approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@lennartkats-db @fjakobs @codecov-commenter @pietern @PaulCornellDB @juliacrawf-db