Create redshift limit one macro (#95)

* Create redshift limit one macro * PR fixes * PR fixes * MagicBot/add-model-counts updates (#94) Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * changelog/readme * changelog --------- Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com>
fivetran · Jan 22, 2025 · 9456612 · 9456612
1 parent 5963347
commit 9456612
Show file tree

Hide file tree

Showing 43 changed files with 235 additions and 58 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,14 @@
+# dbt_shopify_source v0.15.0
+This release includes the following updates:
+
+## Under the Hood 
+- (Affects Redshift only) Creates new `shopify_union_data` macro to accommodate Redshift's treatment of empty tables. ([PR #95](https://github.com/fivetran/dbt_shopify_source/pull/95))
+  - For each staging model, if the source table is not found in any of your schemas, the package will create a table with one row with null values for Redshift destinations. There will be no change in behavior in non-Redshift warehouses.
+  - This is necessary as Redshift will ignore explicit data casts when a table is completely empty and materialize every column as a `varchar`. This throws errors in downstream transformations in the `shopify` package. The 1 row will ensure that Redshift will respect the package's datatype casts.
+
+## Documentation
+- Corrected references to connectors and connections in the README. ([#94](https://github.com/fivetran/dbt_shopify_source/pull/94))
+
 # dbt_shopify_source v0.14.0
 [PR #93](https://github.com/fivetran/dbt_shopify_source/pull/93) includes the following changes:
 

diff --git a/README.md b/README.md
@@ -28,7 +28,7 @@
 ## How do I use the dbt package?
 ### Step 1: Prerequisites
 To use this dbt package, you must have the following:
-- At least one Fivetran Shopify connector syncing data into your destination.
+- At least one Fivetran Shopify connection syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **Databricks**, or **PostgreSQL** destination.
 
 #### Databricks dispatch configuration
@@ -45,11 +45,11 @@ If you  are **not** using the [Shopify transformation package](https://github.co
 ```yml
 packages:
   - package: fivetran/shopify_source
-    version: [">=0.14.0", "<0.15.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.15.0", "<0.16.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ### Step 3: Define database and schema variables
-#### Single connector
+#### Single connection
 By default, this package runs using your destination and the `shopify` schema. If this is not where your Shopify data is (for example, if your Shopify schema is named `shopify_fivetran` and your `issue` table is named `usa_issue`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
@@ -58,8 +58,8 @@ vars:
     shopify_schema: your_schema_name 
 ```
 
-#### Union multiple connectors
-If you have multiple Shopify connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `shopify_union_schemas` OR `shopify_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
+#### Union multiple connections
+If you have multiple Shopify connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `shopify_union_schemas` OR `shopify_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
 
 ```yml
 # dbt_project.yml
@@ -75,7 +75,7 @@ To connect your multiple schema/database sources to the package models, follow t
 
 ### Step 4: Disable models for non-existent sources
 
-The package takes into consideration that not every Shopify connector may have the `fulfillment_event`, `metadata`, or `abandoned_checkout` tables (including `abandoned_checkout`, `abandoned_checkout_discount_code`, and `abandoned_checkout_shipping_line`) and allows you to enable or disable the corresponding functionality. To enable/disable the modeling of the mentioned source tables and their downstream references, add the following variable to your `dbt_project.yml` file:
+The package takes into consideration that not every Shopify connection may have the `fulfillment_event`, `metadata`, or `abandoned_checkout` tables (including `abandoned_checkout`, `abandoned_checkout_discount_code`, and `abandoned_checkout_shipping_line`) and allows you to enable or disable the corresponding functionality. To enable/disable the modeling of the mentioned source tables and their downstream references, add the following variable to your `dbt_project.yml` file:
 
 ```yml
 # dbt_project.yml
@@ -142,8 +142,10 @@ models:
     +schema: my_new_schema_name # leave blank for just the target_schema
 ```
 
-#### Change the source table references (not available if unioning multiple Shopify connectors)
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+
+#### Change the source table references (not available if unioning multiple Shopify connections)
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable: This config is available only when running the package on a single connection:
+
 > IMPORTANT: See this project's [`src_shopify.yml`](https://github.com/fivetran/dbt_shopify_source/blob/main/models/src_shopify.yml) for the default names.
 
 ```yml
@@ -159,7 +161,7 @@ If you are making use of the `shopify_union_schemas` or `shopify_union_databases
 
 Empty staging models are created in the Shopify schema dynamically if the respective source tables do not exist in your raw source schema. For example, if your shop has not incurred any refunds, you will not have a `refund` table yet until you do refund an order, and the package will create an empty `stg_shopify__refund` model.
 
-The source package will will return **completely** empty staging models (ie `limit 0`) if these source tables do not exist in your Shopify schema yet, and the transform package will work seamlessly with these empty models. Once an anticipated source table exists in your schema, the source and transform packages will automatically reference the new populated table(s). ([example](https://github.com/fivetran/dbt_shopify_source/blob/main/models/tmp/stg_shopify__refund_tmp.sql)).
+The source package will will return **completely** empty staging models (ie `limit 0`), if these source tables do not exist in your Shopify schema yet, and the transform package will work seamlessly with these empty models (The one exception is Redshift, which will create 1 all-null row to respect datatype casts). Once an anticipated source table exists in your schema, the source and transform packages will automatically reference the new populated table(s). ([example](https://github.com/fivetran/dbt_shopify_source/blob/main/models/tmp/stg_shopify__refund_tmp.sql)).
 
 The package will raise a compiler warning message that the respective staging model is empty. The compiler warning can be turned off by the end user by setting the `fivetran__remove_empty_table_warnings` variable to `True`.
 

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'shopify_source'
-version: '0.14.0'
+version: '0.15.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'shopify_source_integration_tests'
-version: '0.14.0'
+version: '0.15.0'
 profile: 'integration_tests'
 config-version: 2
 

diff --git a/integration_tests/packages.yml b/integration_tests/packages.yml
@@ -1,2 +1,2 @@
 packages:
-  - local: ../
+  - local: ../
diff --git a/macros/shopify_union_data.sql b/macros/shopify_union_data.sql
@@ -0,0 +1,137 @@
+{%- macro shopify_union_data(table_identifier, database_variable, schema_variable, default_database, default_schema, default_variable, union_schema_variable='union_schemas', union_database_variable='union_databases') -%}
+
+{{ adapter.dispatch('shopify_union_data', 'shopify_source') (
+    table_identifier, 
+    database_variable, 
+    schema_variable, 
+    default_database, 
+    default_schema,
+    default_variable,
+    union_schema_variable,
+    union_database_variable
+    ) }}
+
+{%- endmacro -%}
+
+{%- macro default__shopify_union_data(
+    table_identifier, 
+    database_variable, 
+    schema_variable, 
+    default_database, 
+    default_schema, 
+    default_variable,
+    union_schema_variable,
+    union_database_variable
+    ) -%}
+
+{%- if var(union_schema_variable, none) -%}
+
+    {%- set relations = [] -%}
+
+    {%- if var(union_schema_variable) is string -%}
+    {%- set trimmed = var(union_schema_variable)|trim('[')|trim(']') -%}
+    {%- set schemas = trimmed.split(',')|map('trim'," ")|map('trim','"')|map('trim',"'") -%}
+    {%- else -%}
+    {%- set schemas = var(union_schema_variable) -%}
+    {%- endif -%}
+
+    {%- for schema in var(union_schema_variable) -%}
+    {%- set relation=adapter.get_relation(
+        database=source(schema, table_identifier).database if var('has_defined_sources', false) else var(database_variable, default_database),
+        schema=source(schema, table_identifier).schema if var('has_defined_sources', false) else schema,
+        identifier=source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier
+    ) -%}
+
+    {%- set relation_exists=relation is not none -%}
+
+    {%- if relation_exists -%}
+        {%- do relations.append(relation) -%}
+    {%- endif -%}
+
+    {%- endfor -%}
+
+    {%- if relations != [] -%}
+        {{ dbt_utils.union_relations(relations) }}
+    {%- else -%}
+    {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%}
+    {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }}
+    {% endif -%}
+    select 
+        cast(null as {{ dbt.type_string() }}) as _dbt_source_relation
+    limit {{ '0' if target.type != 'redshift' else '1' }}
+    {%- endif -%}
+
+{%- elif var(union_database_variable, none) -%}
+
+    {%- set relations = [] -%}
+
+    {%- for database in var(union_database_variable) -%}
+    {%- set relation=adapter.get_relation(
+        database=source(schema, table_identifier).database if var('has_defined_sources', false) else database,
+        schema=source(schema, table_identifier).schema if var('has_defined_sources', false) else var(schema_variable, default_schema),
+        identifier=source(schema, table_identifier).identifier if var('has_defined_sources', false) else table_identifier
+    ) -%}
+
+    {%- set relation_exists=relation is not none -%}
+
+    {%- if relation_exists -%}
+        {%- do relations.append(relation) -%}
+    {%- endif -%}
+
+    {%- endfor -%}
+
+    {%- if relations != [] -%}
+        {{ dbt_utils.union_relations(relations) }}
+    {%- else -%}
+    {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%}
+    {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }}
+    {% endif -%}
+    select 
+        cast(null as {{ dbt.type_string() }}) as _dbt_source_relation
+    limit {{ '0' if target.type != 'redshift' else '1' }}
+    {%- endif -%}
+
+{%- else -%}
+    {% set exception_schemas = {"linkedin_company_pages": "linkedin_pages", "instagram_business_pages": "instagram_business"} %}
+    {% set relation = namespace(value="") %}
+    {% if default_schema in exception_schemas.keys() %}
+        {% for corrected_schema_name in exception_schemas.items() %}   
+            {% if default_schema in corrected_schema_name %}
+                {# In order for this macro to effectively work within upstream integration tests (mainly used by the Fivetran dbt package maintainers), this identifier variable selection is required to use the macro with different identifier names. #}
+                {% set identifier_var = corrected_schema_name[1] + "_" + table_identifier + "_identifier"  %}
+                {%- set relation.value=adapter.get_relation(
+                    database=source(corrected_schema_name[1], table_identifier).database,
+                    schema=source(corrected_schema_name[1], table_identifier).schema,
+                    identifier=var(identifier_var, table_identifier)
+                ) -%}
+            {% endif %}
+        {% endfor %}
+    {% else %}
+        {# In order for this macro to effectively work within upstream integration tests (mainly used by the Fivetran dbt package maintainers), this identifier variable selection is required to use the macro with different identifier names. #}
+        {% set identifier_var = default_schema + "_" + table_identifier + "_identifier"  %}
+        {# Unfortunately the Twitter Organic identifiers were misspelled. As such, we will need to account for this in the model. This will be adjusted in the Twitter Organic package, but to ensure backwards compatibility, this needs to be included. #}
+        {% if var(identifier_var, none) is none %} 
+            {% set identifier_var = default_schema + "_" + table_identifier + "_identifer"  %}
+        {% endif %}
+        {%- set relation.value=adapter.get_relation(
+            database=source(default_schema, table_identifier).database,
+            schema=source(default_schema, table_identifier).schema,
+            identifier=var(identifier_var, table_identifier)
+        ) -%}
+    {% endif %}
+{%- set table_exists=relation.value is not none -%}
+
+{%- if table_exists -%}
+    select * 
+    from {{ relation.value }}
+{%- else -%}
+    {% if execute and not var('fivetran__remove_empty_table_warnings', false) -%}
+    {{ exceptions.warn("\n\nPlease be aware: The " ~ table_identifier|upper ~ " table was not found in your " ~ default_schema|upper ~ " schema(s). The Fivetran dbt package will create a completely empty " ~ table_identifier|upper ~ " staging model as to not break downstream transformations. To turn off these warnings, set the `fivetran__remove_empty_table_warnings` variable to TRUE (see https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source for details).\n") }}
+    {% endif -%}
+    select 
+        cast(null as {{ dbt.type_string() }}) as _dbt_source_relation
+    limit {{ '0' if target.type != 'redshift' else '1' }}
+{%- endif -%}
+{%- endif -%}
+
+{%- endmacro -%}
diff --git a/models/tmp/stg_shopify__abandoned_checkout_discount_code_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_discount_code_tmp.sql
@@ -1,7 +1,7 @@
 {{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='abandoned_checkout_discount_code', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__abandoned_checkout_shipping_line_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_shipping_line_tmp.sql
@@ -1,7 +1,7 @@
 {{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='abandoned_checkout_shipping_line', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__abandoned_checkout_tmp.sql b/models/tmp/stg_shopify__abandoned_checkout_tmp.sql
@@ -1,7 +1,7 @@
 {{ config(enabled=var('shopify_using_abandoned_checkout', True)) }}
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='abandoned_checkout', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__collection_product_tmp.sql b/models/tmp/stg_shopify__collection_product_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='collection_product', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__collection_tmp.sql b/models/tmp/stg_shopify__collection_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='collection', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__customer_tag_tmp.sql b/models/tmp/stg_shopify__customer_tag_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='customer_tag', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__customer_tmp.sql b/models/tmp/stg_shopify__customer_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='customer', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__discount_code_tmp.sql b/models/tmp/stg_shopify__discount_code_tmp.sql
@@ -1,7 +1,7 @@
 -- this model will be all NULL until you create a discount code in Shopify
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='discount_code', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__fulfillment_event_tmp.sql b/models/tmp/stg_shopify__fulfillment_event_tmp.sql
@@ -1,7 +1,7 @@
 {{ config(enabled=var('shopify_using_fulfillment_event', false)) }}
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='fulfillment_event', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__fulfillment_tmp.sql b/models/tmp/stg_shopify__fulfillment_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='fulfillment', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__inventory_item_tmp.sql b/models/tmp/stg_shopify__inventory_item_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='inventory_item', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__inventory_level_tmp.sql b/models/tmp/stg_shopify__inventory_level_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='inventory_level', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__location_tmp.sql b/models/tmp/stg_shopify__location_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='location', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__metafield_tmp.sql b/models/tmp/stg_shopify__metafield_tmp.sql
@@ -1,7 +1,7 @@
 {{ config(enabled=var('shopify_using_metafield', True)) }}
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='metafield', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_adjustment_tmp.sql b/models/tmp/stg_shopify__order_adjustment_tmp.sql
@@ -1,7 +1,7 @@
 -- this model will be all NULL until you have made an order adjustment in Shopify
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='order_adjustment', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_discount_code_tmp.sql b/models/tmp/stg_shopify__order_discount_code_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='order_discount_code', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_line_refund_tmp.sql b/models/tmp/stg_shopify__order_line_refund_tmp.sql
@@ -1,7 +1,7 @@
 -- this model will be all NULL until you have made an order line refund in Shopify
 
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='order_line_refund', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_line_tmp.sql b/models/tmp/stg_shopify__order_line_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+   shopify_source.shopify_union_data(
         table_identifier='order_line', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_note_attribute_tmp.sql b/models/tmp/stg_shopify__order_note_attribute_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='order_note_attribute', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema', 

diff --git a/models/tmp/stg_shopify__order_shipping_line_tmp.sql b/models/tmp/stg_shopify__order_shipping_line_tmp.sql
@@ -1,5 +1,5 @@
 {{
-    fivetran_utils.union_data(
+    shopify_source.shopify_union_data(
         table_identifier='order_shipping_line', 
         database_variable='shopify_database', 
         schema_variable='shopify_schema',