Merge pull request #2646 from dbt-labs/core-1-dot-4

Updates to upgrade guides for 1.4

website/docs/docs/build/exposures.md

@@ -24,6 +24,40 @@ Exposures make it possible to define and describe a downstream use of your dbt p
 
 Exposures are defined in `.yml` files nested under an `exposures:` key.
 
+<VersionBlock firstVersion="1.4">
+
+<File name='models/<filename>.yml'>
+
+```yaml
+version: 2
+
+exposures:
+
+  - name: weekly_jaffle_metrics
+    label: Jaffles by the Week
+    type: dashboard
+    maturity: high
+    url: https://bi.tool/dashboards/1
+    description: >
+      Did someone say "exponential growth"?
+
+    depends_on:
+      - ref('fct_orders')
+      - ref('dim_customers')
+      - source('gsheets', 'goals')
+      - metric('count_orders')
+
+    owner:
+      name: Callum McData
+      email: data@jaffleshop.com
+```
+
+</File>
+
+</VersionBlock>
+
+<VersionBlock lastVersion="1.3">
+
 <File name='models/<filename>.yml'>
 
 ```yaml
@@ -50,15 +84,28 @@ exposures:
 
 </File>
 
+</VersionBlock>
+
 ### Available properties
 
 _Required:_
-- **name** (must be unique among exposures and you must use the [snake case](https://en.wikipedia.org/wiki/Snake_case) naming convention)
+- **name**: a unique exposure name written in [snake case](https://en.wikipedia.org/wiki/Snake_case)
 - **type**: one of `dashboard`, `notebook`, `analysis`, `ml`, `application` (used to organize in docs site)
 - **owner**: email
 
+<VersionBlock firstVersion="1.4">
+
+_Expected:_
+- **depends_on**: list of refable nodes, including `ref`, `source`, and `metric` (While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly)
+
+</VersionBlock>
+
+<VersionBlock lastVersion="1.3">
+
 _Expected:_
-- **depends_on**: list of refable nodes (`ref` + `source`)
+- **depends_on**: list of refable nodes, including `ref` and `source` (While possible, it is highly unlikely you will ever need an `exposure` to depend on a `source` directly)
+
+</VersionBlock>
 
 _Optional:_
 - **url**:  enables the link to **View this exposure** in the upper right corner of the generated documentation site

website/docs/docs/build/packages.md

@@ -135,6 +135,14 @@ Add the Git URL for the package, and optionally specify a revision. The revision
 - a tagged release
 - a specific commit (full 40-character hash)
 
+Example of a revision specifying a 40-character hash:
+
+```yaml
+packages:
+  - git: "https://github.com/dbt-labs/dbt-utils.git"
+    revision: 4e28d6da126e2940d17f697de783a717f2503188
+```
+
 We **strongly recommend** "pinning" your package to a specific release by specifying a release name.
 
 If you do not provide a revision, or if you use `master`, then any updates to the package will be incorporated into your project the next time you run `dbt deps`. While we generally try to avoid making breaking changes to these packages, they are sometimes unavoidable. Pinning a package revision helps prevent your code from changing without your explicit approval.
@@ -143,6 +151,23 @@ To find the latest release for a package, navigate to the `Releases` tab in the
 
 As of v0.14.0, dbt will warn you if you install a package using the `git` syntax without specifying a version (see below).
 
+<VersionBlock firstVersion="1.4">
+
+### Internally hosted tarball URL
+
+Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments such as Artifactory or cloud storage buckets, dbt Core enables you to install packages from internally-hosted tarball URLs. 
+
+
+```yaml
+packages:
+  - tarball: https://codeload.github.com/dbt-labs/dbt-utils/tar.gz/0.9.6
+    name: 'dbt_utils'
+```
+
+Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within.
+
+</VersionBlock>
+
 ### Private packages
 
 #### SSH Key Method (Command Line only)

website/docs/docs/build/python-models.md

@@ -569,7 +569,7 @@ In their initial launch, Python models are supported on three of the most popula
 
 **Installing packages:** Snowpark supports several popular packages via Anaconda. The complete list is at https://repo.anaconda.com/pkgs/snowflake/. Packages are installed at the time your model is being run. Different models can have different package dependencies. If you are using third-party packages, Snowflake recommends using a dedicated virtual warehouse for best performance rather than one with many concurrent users.
 
-**About "sprocs":** dbt submits Python models to run as _stored procedures_, which some people call _sprocs_ for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then _call_ it to execute. Snowpark has a Public Preview feature for _temporary_ or _anonymous_ stored procedures ([docs](https://docs.snowflake.com/en/sql-reference/sql/call-with.html)), which are faster and leave a cleaner query history. You can switch this feature on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models in a future release.
+**About "sprocs":** dbt submits Python models to run as _stored procedures_, which some people call _sprocs_ for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then _call_ it to execute. Snowpark has an Open Preview feature for _temporary_ or _anonymous_ stored procedures ([docs](https://docs.snowflake.com/en/sql-reference/sql/call-with.html)), which are faster and leave a cleaner query history. You can switch this feature on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models starting with the release of dbt Core version 1.4.
 
 <File name='dbt_project.yml'>
 

website/docs/faqs/Core/install-python-compatibility.md

@@ -5,9 +5,21 @@ sidebar_label: 'Python version'
 id: install-python-compatibility
 ---
 
-<VersionBlock firstVersion="1.1">
+<VersionBlock firstVersion="1.4">
 
-The latest version of `dbt-core` is compatible with Python versions 3.7, 3.8, 3.9, and 3.10.
+The latest version of `dbt-core` is compatible with Python versions 3.7, 3.8, 3.9, 3.10, and 3.11.
+
+:::info
+
+Full support of Python 3.11 across adapters is a work in progress since adapter plugins have additional third-party dependencies. As of January 2023, dbt-snowflake does not yet support Python 3.11 (relevant issue in snowflake-connector-python).
+
+:::
+
+</VersionBlock>
+
+<VersionBlock firstVersion="1.1" lastVersion="1.3">
+
+The latest version of `dbt-core` is compatible with Python versions 3.7, 3.8, 3.9, and 3.10
 
 </VersionBlock>
 

website/docs/guides/migration/versions/04-upgrading-to-v1.4.md

@@ -0,0 +1,46 @@
+---
+title: "Upgrading to v1.4 (prerelease)"
+description: New features and changes in dbt Core v1.4
+---
+### Resources
+
+- [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
+- [CLI Installation guide](/docs/get-started/installation)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+
+## What to know before upgrading
+
+dbt Labs is committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).
+
+### For consumers of dbt artifacts (metadata)
+
+The manifest schema version has updated to `v8`.
+These changes are relevant for people who:
+- parse or analyze the contents of the `manifest.json` file
+- have custom code accessing the [`model`](https://docs.getdbt.com/reference/dbt-jinja-functions/model) or [`graph`](https://docs.getdbt.com/reference/dbt-jinja-functions/graph) variables. For example:
+
+```
+{{ model.root_path }}
+```
+
+Relevant changes are: 
+- The `root_path` attribute is removed for all nodes to reduce duplicative information.
+- Unused attributes have been removed from seed `nodes`, including `depends_on`, and from `macros`, including `tags`.
+- The `unique_id` of docs blocks now start with `doc` for consistency with other resource types.
+
+## New and changed documentation
+
+- [Python 3.11](/faqs/Core/install-python-compatibility) was released in October 2022. It is officially supported in dbt-core v1.4, though full support depends also on your data platform adapter plugin.
+According to the Python maintainers, "Python 3.11 is between 10-60% faster than Python 3.10." We encourage you to try a `dbt parse` with dbt Core v1.4 + Python 3.11, and compare it to the result of dbt parse on dbt Core v1.3 + Python 3.10.
+- Some organizations have security requirements to pull resources only from internal services. To address the need to install packages from hosted environments (such as Artifactory or cloud storage buckets), packages now support [installing from internally hosted tarball URLs](/docs/build/packages#Internally-hosted-tarball-URL). 
+- The [local_md5](/reference/dbt-jinja-functions/local-md5) context variable is a new Jinja function that calculates an [MD5 hash](https://en.wikipedia.org/wiki/MD5).
+- [Exposures](/docs/build/exposures) can now depend on `metrics`.
+- [Coming soon] `--favor-state` ([dbt-labs/docs.getdbt.com/issues/2021](https://github.com/dbt-labs/docs.getdbt.com/issues/2021))
+- [Coming soon] `incremental_predicates` ([dbt-labs/docs.getdbt.com/issues/2636](https://github.com/dbt-labs/docs.getdbt.com/issues/2636))
+- [Coming soon] BigQuery: `time_ingestion_partitioning` + `insert_overwrite` ([dbt-labs/docs.getdbt.com/issues/2426](https://github.com/dbt-labs/docs.getdbt.com/issues/2426))
+
+
+### Updates to Python models
+
+- [Stored procedures](/docs/build/python-models##Specific-data-platforms) for Python models will be enabled for all dbt + Snowpark Python models starting with the release of dbt Core version 1.4.
+

website/docs/guides/migration/versions/05-upgrading-to-v1.3.md

@@ -1,13 +1,14 @@
 ---
 title: "Upgrading to v1.3 (latest)"
+description: New features and changes in dbt Core v1.3
 ---
 ### Resources
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/get-started/installation)
 - [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
 
-## Breaking changes
+## What to know before upgrading
 
 We are committed to providing backward compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).
 

website/docs/guides/migration/versions/06-upgrading-to-v1.2.md

@@ -1,13 +1,14 @@
 ---
 title: "Upgrading to v1.2"
+description: New features and changes in dbt Core v1.2
 ---
 ### Resources
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.2.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/get-started/installation)
 - [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
 
-## Breaking changes
+## What to know before upgrading
 
 There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).
 

website/docs/guides/migration/versions/07-upgrading-to-v1.1.md

@@ -1,13 +1,14 @@
 ---
 title: "Upgrading to v1.1"
+description: New features and changes in dbt Core v1.1
 ---
 ### Resources
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.1.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/get-started/installation)
 - [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
 
-## Breaking changes
+## What to know before upgrading
 
 There are no breaking changes for code in dbt projects and packages. We are committed to providing backwards compatibility for all versions 1.x. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).
 

website/docs/guides/migration/versions/08-upgrading-to-v1.0.md

@@ -1,5 +1,6 @@
 ---
 title: "Upgrading to v1.0"
+description: New features and changes in dbt Core v1.0
 ---
 ### Resources
 
@@ -8,7 +9,7 @@ title: "Upgrading to v1.0"
 - [CLI Installation guide](/docs/get-started/installation)
 - [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
 
-## Breaking changes
+## What to know before upgrading
 
 dbt Core major version 1.0 includes a number of breaking changes! Wherever possible, we have offered backwards compatibility for old behavior, and (where necessary) made migration simple.
 

website/docs/reference/artifacts/manifest-json.md

@@ -1,8 +1,25 @@
 ---
 title: Manifest
 ---
+<VersionBlock firstVersion="1.4">
+**dbt Core v1.4 produces schema**: [`v8`](https://schemas.getdbt.com/dbt/manifest/v8/index.html)
+</VersionBlock>
 
-**Current schema**: [`v6`](https://schemas.getdbt.com/dbt/manifest/v6/index.html)
+<VersionBlock lastVersion="1.3">
+**dbt Core v1.3 produces schema**: [`v7`](https://schemas.getdbt.com/dbt/manifest/v7/index.html)
+</VersionBlock>
+
+<VersionBlock lastVersion="1.2">
+**dbt Core v1.2 produces schema**: [`v6`](https://schemas.getdbt.com/dbt/manifest/v6/index.html)
+</VersionBlock>
+
+<VersionBlock lastVersion="1.1">
+**dbt Core v1.1 produces schema**: [`v5`](https://schemas.getdbt.com/dbt/manifest/v5/index.html)
+</VersionBlock>
+
+<VersionBlock lastVersion="1.0">
+**dbt Core v1.0 produces schema**: [`v4`](https://schemas.getdbt.com/dbt/manifest/v4/index.html)
+</VersionBlock>
 
 **Produced by:** [`build`](commands/build) [`compile`](commands/compile) [`docs generate`](commands/cmd-docs) [`list`](commands/list) [`seed`](commands/seed) [`snapshot`](commands/snapshot) [`source freshness`](commands/source) [`test`](commands/test) [`run`](commands/run) [`run-operation`](commands/run-operation)
 
@@ -31,7 +48,7 @@ All resources nested within `nodes`, `sources`, `exposures`, `macros`, and `docs
 - `name`: Resource name.
 - `unique_id`: `<resource_type>.<package>.<resource_name>`, same as dictionary key
 - `package_name`: Name of package that defines this resource.
-- `root_path`: Absolute file path of this resource's package.
+- `root_path`: Absolute file path of this resource's package. This is removed in dbt Core v1.4 / manifest v8 to reduce duplicative information across nodes.
 - `path`: Relative file path of this resource's definition within its "resource path" (`model-paths`, `seed-paths`, etc.).
 - `original_file_path`: Relative file path of this resource's definition, including its resource path.
 

website/docs/reference/dbt-jinja-functions/local-md5.md

@@ -0,0 +1,18 @@
+---
+title: "local_md5"
+id: "local_md5"
+---
+
+The `local_md5` context variable calculates an [MD5 hash](https://en.wikipedia.org/wiki/MD5) of the given string. The string `local_md5` emphasizes that the hash is calculated _locally_, in the dbt-Jinja context. This variable is typically useful for advanced use cases. For example, when you generate unique identifiers within custom materialization or operational logic, you can either avoid collisions between temporary relations or identify changes by comparing checksums.
+
+It is different than the `md5` SQL function, supported by many SQL dialects, which runs remotely in the data platform. You want to always use SQL hashing functions when generating <Term id="surrogate-key">surrogate keys</Term>.
+
+Usage:
+```sql
+-- source
+{%- set value_hash = local_md5("hello world") -%}
+'{{ value_hash }}'
+
+-- compiled
+'5eb63bbbe01eeed093cb22bb8f5acdc3'
+```