Merge pull request #47 from fivetran/feature/revamp/transform-enhance…

…ments

transform work, some documentation

CHANGELOG.md

@@ -2,6 +2,41 @@
 ## 🎉 Documentation and Feature Updates
 - Updated README documentation updates for easier navigation and setup of the dbt package
 - Included `shopify_[source_table_name]_identifier` variable for additional flexibility within the package when source tables are named differently.
+- The following fields have been added to their respective staging models:
+  - `stg_shopify__order`:
+    - `total_discounts_set`
+    - `total_line_items_price_set`
+    - `total_price_usd`
+    - `total_price_set`
+    - `total_tax_set`
+    - `total_tip_received`
+    - `is_deleted`
+    - `app_id`
+    - `checkout_id`
+    - `client_details_user_agent`
+    - `customer_locale`
+    - `order_status_url`
+    - `presentment_currency`
+    - `is_inventory_confirmed`
+  - `stg_shopify__customer`:
+    - `note`
+    - `lifetime_duration`
+    - `currency`
+    - `marketing_consent_state` (coalescing of `email_marketing_consent_state` and deprecated `accepts_marketing` field)
+    - `marketing_opt_in_level` (coalescing of `email_marketing_consent_opt_in_level` and deprecated `marketing_opt_in_level` field)
+    - `marketing_consent_updated_at` (coalescing of `email_marketing_consent_consent_updated_at` and deprecated `accepts_marketing_updated_at` field)
+  - `stg_shopify__order_line_refund`:
+    - `subtotal_set`
+    - `total_tax_set`
+  - `stg_shopify__order_line`:
+    - `pre_tax_price_set`
+    - `price_set`
+    - `tax_code`
+    - `total_discount_set`
+    - `variant_title`
+    - `variant_inventory_management`
+    - `vendor`
+    - `properties`
 
 ## 🚨 Breaking Changes 🚨:
 [PR #36](https://github.com/fivetran/dbt_shopify_source/pull/36) includes the following breaking changes:
@@ -39,6 +74,8 @@
     - `dbt.current_timestamp_backcompat`
     - `dbt.current_timestamp_in_utc_backcompat`
 - Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
+- The following fields have been removed as they have been deprecated:
+  - 
 
 [PR #40](https://github.com/fivetran/dbt_shopify_source/pull/40) includes the following breaking changes:
 - The `is_requiring_shipping` field in `stg_shopify__order_line` has been renamed to `is_shipping_required`, just because it sounds better :)

DECISIONLOG.md

@@ -12,4 +12,13 @@ Thus, the source package will create empty (1 row of all `NULL` fields) staging
 
 ## Keeping Deleted Entities 
 
-todo - not filtering out _fivetran_deleted in staging models. when joining these tables together in the transform package, bring in _fivetran_deleted as is_<foreign key table>_deleted
+todo - not filtering out _fivetran_deleted in staging models. when joining these tables together in the transform package, bring in _fivetran_deleted as is_<foreign key table>_deleted
+
+## Accepted Value Test Severity
+
+We test the following columns for accepted values because their values are hard-coded to be pivoted out into columns and/or used as `JOIN` conditions in downstream models.
+- `stg_shopify__price_rule.target_type`: accepted values are `line_item`, `shipping_line`
+- `stg_shopify__price_rule.value_type`: accepted values are `percentage`, `fixed_amount`
+- `stg_shopify__fulfillment.status`: accepted values are `pending`, `open`, `success`, `cancelled`, `error`, `failure`
+
+We have chosen to make the severity of these tests `warn`, as non-accepted values will be filtered out in the transformation models. They will not introduce erroneous data.

README.md

@@ -25,14 +25,15 @@ To use this dbt package, you must have the following:
 - At least one Fivetran Shopify connector syncing data into your destination. 
 - A **BigQuery**, **Snowflake**, **Redshift**, **Databricks**, or **PostgreSQL** destination.
 
-## Step 2: Install the package
-Include the following shopify_source package version in your `packages.yml` file.
+## Step 2: Install the package (skip if also using the `dbt_shopify` package)
+If you  are **not** using the [Shopify transformation package](https://github.com/fivetran/dbt_shopify), include the following package version in your `packages.yml` file. If you are installing the transform package, the source package is automatically installed as a dependency.
 > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 ```yml
 packages:
   - package: fivetran/shopify_source
     version: [">=0.7.0", "<0.8.0"]
 ```
+
 ## Step 3: Define database and schema variables
 ### Single connector
 By default, this package runs using your destination and the `shopify` schema. If this is not where your Shopify data is (for example, if your Shopify schema is named `shopify_fivetran` and your `issue` table is named `usa_issue`), add the following configuration to your root `dbt_project.yml` file:
@@ -59,20 +60,31 @@ vars:
 ## (Optional) Step 5: Additional configurations
 <details><summary>Expand to view configurations</summary>
 
-### Add Passthrough Columns - todo, update with new syntax
-This package includes all source columns defined in the [staging_columns.sql](https://github.com/fivetran/dbt_shopify_source/blob/master/macros/staging_columns.sql) macro. To add additional columns to this package, do so using our pass-through column variables in your root `dbt_project.yml`. This is extremely useful if you'd like to include custom fields to the package.
+### Passing Through Additional Fields
+This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
 
 ```yml
 # dbt_project.yml
 
 vars:
   shopify_source:
-    customer_pass_through_columns: []
-    order_line_refund_pass_through_columns: []
-    order_line_pass_through_columns: []
-    order_pass_through_columns: []
-    product_pass_through_columns: []
-    product_variant_pass_through_columns: []
+    customer_pass_through_columns:
+      - name: "customer_custom_field"
+        alias: "customer_field"
+    order_line_refund_pass_through_columns:
+      - name: "unique_string_field"
+        alias: "field_id"
+        transform_sql: "cast(field_id as string)"
+    order_line_pass_through_columns:
+      - name: "that_field"
+    order_pass_through_columns:
+      - name: "sub_field"
+        alias: "subsidiary_field"
+    product_pass_through_columns:
+      - name: "this_field"
+    product_variant_pass_through_columns:
+      - name: "new_custom_field"
+        alias: "custom_field"
 ```
 
 ### Changing the Build Schema

dbt_project.yml

@@ -42,6 +42,7 @@ vars:
     tender_transaction_source: "{{ source('shopify','tender_transaction') }}"
     abandoned_checkout_discount_code_source: "{{ source('shopify','abandoned_checkout_discount_code') }}" 
     order_discount_code_source: "{{ source('shopify','order_discount_code') }}" 
+    abandoned_checkout_shipping_line_source: "{{ source('shopify', 'abandoned_checkout_shipping_line') }}"
 
     customer_pass_through_columns: []
     order_line_refund_pass_through_columns: []

integration_tests/dbt_project.yml

@@ -4,6 +4,7 @@ profile: 'integration_tests'
 config-version: 2
 
 vars:
+  shopify_schema: shopify_source_integration_tests_2
   shopify_source:
     shopify_customer_identifier:          "shopify_customer_data"
     shopify_order_line_refund_identifier: "shopify_order_line_refund_data"
@@ -14,30 +15,29 @@ vars:
     shopify_product_variant_identifier:   "shopify_product_variant_data"
     shopify_refund_identifier:            "shopify_refund_data"
     shopify_transaction_identifier:       "shopify_transaction_data"
-    shopify_schema: shopify_source_integration_tests_2
-
-    abandoned_checkout_source: "{{ ref('shopify_abandoned_checkout_data') }}"
-    collection_product_source: "{{ ref('shopify_collection_product_data') }}"
-    collection_source: "{{ ref('shopify_collection_data') }}"
-    customer_tag_source: "{{ ref('shopify_customer_tag_data') }}"
-    discount_code_source: "{{ ref('shopify_discount_code_data') }}"
-    fulfillment_source: "{{ ref('shopify_fulfillment_data') }}"
-    inventory_item_source: "{{ ref('shopify_inventory_item_data') }}"
-    inventory_level_source: "{{ ref('shopify_inventory_level_data') }}"
-    location_source: "{{ ref('shopify_location_data') }}"
-    metafield_source: "{{ ref('shopify_metafield_data') }}"
-    order_note_attribute_source: "{{ ref('shopify_order_note_attribute_data') }}"
-    order_shipping_line_source: "{{ ref('shopify_order_shipping_line_data') }}"
-    order_shipping_tax_line_source: "{{ ref('shopify_order_shipping_tax_line_data') }}"
-    order_tag_source: "{{ ref('shopify_order_tag_data') }}"
-    order_url_tag_source: "{{ ref('shopify_order_url_tag_data') }}"
-    price_rule_source: "{{ ref('shopify_price_rule_data') }}"
-    product_image_source: "{{ ref('shopify_product_image_data') }}"
-    product_tag_source: "{{ ref('shopify_product_tag_data') }}"
-    shop_source: "{{ ref('shopify_shop_data') }}"
-    tender_transaction_source: "{{ ref('shopify_tender_transaction_data') }}"
-    abandoned_checkout_discount_code_source: "{{ ref('shopify_abandoned_checkout_discount_code_data') }}"
-    order_discount_code_source: "{{ ref('shopify_order_discount_code_data') }}"
+    shopify_abandoned_checkout_identifier: "shopify_abandoned_checkout_data"
+    shopify_collection_product_identifier: "shopify_collection_product_data"
+    shopify_collection_identifier: "shopify_collection_data"
+    shopify_customer_tag_identifier: "shopify_customer_tag_data"
+    shopify_discount_code_identifier: "shopify_discount_code_data"
+    shopify_fulfillment_identifier: "shopify_fulfillment_data"
+    shopify_inventory_item_identifier: "shopify_inventory_item_data"
+    shopify_inventory_level_identifier: "shopify_inventory_level_data"
+    shopify_location_identifier: "shopify_location_data"
+    shopify_metafield_identifier: "shopify_metafield_data"
+    shopify_order_note_attribute_identifier: "shopify_order_note_attribute_data"
+    shopify_order_shipping_line_identifier: "shopify_order_shipping_line_data"
+    shopify_order_shipping_tax_line_identifier: "shopify_order_shipping_tax_line_data"
+    shopify_order_tag_identifier: "shopify_order_tag_data"
+    shopify_order_url_tag_identifier: "shopify_order_url_tag_data"
+    shopify_price_rule_identifier: "shopify_price_rule_data"
+    shopify_product_image_identifier: "shopify_product_image_data"
+    shopify_product_tag_identifier: "shopify_product_tag_data"
+    shopify_shop_identifier: "shopify_shop_data"
+    shopify_tender_transaction_identifier: "shopify_tender_transaction_data"
+    shopify_abandoned_checkout_discount_code_identifier: "shopify_abandoned_checkout_discount_code_data"
+    shopify_order_discount_code_identifier: "shopify_order_discount_code_data"
+    shopify_abandoned_checkout_shipping_line_identifier: "shopify_abandoned_checkout_shipping_line_data"
 
 dispatch:
   - macro_namespace: dbt_utils
@@ -155,4 +155,10 @@ seeds:
     shopify_inventory_item_data:
       +column_types:
         updated_at: timestamp
-        created_at: timestamp
+        created_at: timestamp
+    shopify_abandoned_checkout_shipping_line_data:
+      +column_types:
+        markup: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
+        price: float
+        original_shop_markup: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
+        original_shop_price: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"

integration_tests/seeds/shopify_abandoned_checkout_shipping_line_data.csv

@@ -0,0 +1,6 @@
+checkout_id,index,_fivetran_synced,api_client_id,carrier_identifier,carrier_service_id,code,delivery_category,discounted_price,id,markup,phone,price,requested_fulfillment_service_id,source,title,validation_context,delivery_expectation_range,delivery_expectation_type,original_shop_markup,original_shop_price,presentment_title,delivery_expectation_range_min,delivery_expectation_range_max
+653675,1,2023-01-09 06:48:18.093000,,,,Standard,,,c3ce0972c2e30eaf7001bea,0.0,,0.0,,shopify,Standard,,,,0.0,0.0,Standard,,
+379,1,2023-01-09 06:48:23.540000,,,,Standard,,,bf7c90953344902c13,0.0,,0.0,,shopify,Standard,,,,0.0,0.0,Standard,,
+635,1,2023-01-09 06:48:24.243000,,,,Standard,,,519ff4275cd972e282db,0.0,,0.0,,shopify,Standard,,,,0.0,0.0,Standard,,
+3211,1,2023-01-09 06:48:18.068000,,,,Standard,,,8d18671d481ad46a,0.0,,0.0,,shopify,Standard,,,,0.0,0.0,Standard,,
+381227,1,2023-01-09 06:48:16.985000,,,,Standard,,,8f2fab1b455ec9e597,0.0,,0.0,,shopify,Standard,,,,0.0,0.0,Standard,,

macros/get_abandoned_checkout_shipping_line_columns.sql

@@ -0,0 +1,32 @@
+{% macro get_abandoned_checkout_shipping_line_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "api_client_id", "datatype": dbt.type_int()},
+    {"name": "carrier_identifier", "datatype": dbt.type_string()},
+    {"name": "carrier_service_id", "datatype": dbt.type_int()},
+    {"name": "checkout_id", "datatype": dbt.type_int()},
+    {"name": "code", "datatype": dbt.type_string()},
+    {"name": "delivery_category", "datatype": dbt.type_string()},
+    {"name": "delivery_expectation_range", "datatype": dbt.type_string()},
+    {"name": "delivery_expectation_range_max", "datatype": dbt.type_int()},
+    {"name": "delivery_expectation_range_min", "datatype": dbt.type_int()},
+    {"name": "delivery_expectation_type", "datatype": dbt.type_string()},
+    {"name": "discounted_price", "datatype": dbt.type_float()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "index", "datatype": dbt.type_int()},
+    {"name": "markup", "datatype": dbt.type_float()},
+    {"name": "original_shop_markup", "datatype": dbt.type_string()},
+    {"name": "original_shop_price", "datatype": dbt.type_string()},
+    {"name": "phone", "datatype": dbt.type_string()},
+    {"name": "presentment_title", "datatype": dbt.type_string()},
+    {"name": "price", "datatype": dbt.type_float()},
+    {"name": "requested_fulfillment_service_id", "datatype": dbt.type_string()},
+    {"name": "source", "datatype": dbt.type_string()},
+    {"name": "title", "datatype": dbt.type_string()},
+    {"name": "validation_context", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/get_order_columns.sql

@@ -20,7 +20,6 @@
     {"name": "total_weight", "datatype": dbt.type_numeric()},
     {"name": "total_tip_received", "datatype": dbt.type_float()},
     {"name": "landing_site_base_url", "datatype": dbt.type_string()},
-    {"name": "landing_site_ref", "datatype": dbt.type_string()},
     {"name": "location_id", "datatype": dbt.type_numeric()},
     {"name": "name", "datatype": dbt.type_string()},
     {"name": "note", "datatype": dbt.type_string()},
@@ -80,7 +79,6 @@
     {"name": "checkout_id", "datatype": dbt.type_int()},
     {"name": "client_details_user_agent", "datatype": dbt.type_string()},
     {"name": "customer_locale", "datatype": dbt.type_string()},
-    {"name": "device_id", "datatype": dbt.type_string()},
     {"name": "order_status_url", "datatype": dbt.type_string()},
     {"name": "presentment_currency", "datatype": dbt.type_string()},
     {"name": "confirmed", "datatype": "boolean"}