diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh index 3828b34..3e08693 100644 --- a/.buildkite/scripts/run_models.sh +++ b/.buildkite/scripts/run_models.sh @@ -19,7 +19,5 @@ dbt deps dbt seed --target "$db" --full-refresh dbt run --target "$db" --full-refresh dbt test --target "$db" -dbt run --vars '{shopify__using_order_adjustment: false, shopify__using_order_line_refund: false, shopify__using_refund: false}' --target "$db" --full-refresh -dbt test --target "$db" -dbt run --vars '{shopify__using_order_line_refund: false}' --target "$db" --full-refresh -dbt run --vars '{shopify__using_refund: false}' --target "$db" --full-refresh \ No newline at end of file +dbt run --vars '{shopify_timezone: "America/New_York"}' --target "$db" --full-refresh +dbt test --target "$db" \ No newline at end of file diff --git a/DECISIONLOG.md b/DECISIONLOG.md index 95bebf6..a5e6ae2 100644 --- a/DECISIONLOG.md +++ b/DECISIONLOG.md @@ -1,7 +1,21 @@ ## Decision Log +In creating this package, which is meant for a wide range of use cases, we had to take opinionated stances on a few different questions we came across during development. We've consolidated significant choices we made here, and will continue to update as the package evolves. + ### Refund/Return Timestamp Mismatch In validating metrics with the Sales over Time reports in the Shopify UI, you may detect discrepancies in reported revenue. A known difference between this package's reporting and the Shopify UI is that Shopify's UI will report refunded revenue on the date that the _return was processed_ (see Shopify [docs](https://help.shopify.com/en/manual/reports-and-analytics/shopify-reports/report-types/sales-report)), whereas this package reports on the date the _order was placed_. So, if a customer placed an order amounting to $50 on November 30th, 2022 and fully returned it on December 1st, 2022, the package would report $0 net sales for this customer on November 30th, while Shopify would report $50 in sales on November 30th and -$50 on December 1st. -We felt that reporting on the order date made more sense in reality, but, if you feel differently, please reach out and create a Feature Request. To align with the Shopify method yourself, this would most likely involve aggregating `transactions` data (relying on the `kind` column to determine sales vs returns) instead of `orders`. \ No newline at end of file +We felt that reporting on the order date made more sense in reality, but, if you feel differently, please reach out and create a Feature Request. To align with the Shopify method yourself, this would most likely involve aggregating `transactions` data (relying on the `kind` column to determine sales vs returns) instead of `orders`. + +## Creating Empty Tables for Refunds, Order Line Refunds, and Order Adjustments + +Source tables related to `refunds`, `order_line_refunds`, and `order_adjustments` are created in the Shopify schema dyanmically. For example, if your shop has not incurred any refunds, you will not have a `refund` table yet until you do refund an order. + +Thus, the source package will create empty (1 row of all `NULL` fields) staging models if these source tables do not exist in your Shopify schema yet, and the transform package will work seamlessly with these empty models. Once `refund`, `order_line_refund`, or `order_adjustment` exists in your schema, the source and transform packages will automatically reference the new populated table(s). ([example](https://github.com/fivetran/dbt_shopify_source/blob/main/models/tmp/stg_shopify__refund_tmp.sql)). + +> In previous versions of the package, you had to manually enable or disable transforms of `refund`, `order_line_refund`, or `order_adjustment` through variables. Because this required you to monitor your Shopify account/schema and update the variable(s) accordingly, we decided to pursue a more automated solution. + +## Keeping Deleted Entities + +todo - not filtering out _fivetran_deleted in staging models. when joining these tables together in the transform package, bring in _fivetran_deleted as is__deleted \ No newline at end of file diff --git a/README.md b/README.md index 83d8c82..7274fce 100644 --- a/README.md +++ b/README.md @@ -66,17 +66,7 @@ vars: shopify_union_databases: ['shopify_usa','shopify_canada'] # use this if the data is in different databases/projects but uses the same schema name ``` -## Step 4: Disable models for non-existent sources -This package was designed with the intention that users have all relevant Shopify tables being synced by Fivetran. However, if you are a Shopify user that does not operate on `returns` or `adjustments` then you will not have the related source tables. As such, you may use the below variable configurations to disable the respective downstream models. All variables are `true` by default. Only add the below configuration to your root `dbt_project.yml` if you are wishing to disable the models: - -```yml -# dbt_project.yml - -vars: - shopify__using_order_adjustment: false # true by default - shopify__using_order_line_refund: false # true by default - shopify__using_refund: false # true by default -``` +## Step 4: TODO - timezone converting, maybe should be optional ## (Optional) Step 5: Additional configurations
Expand for configurations diff --git a/dbt_project.yml b/dbt_project.yml index 07d2125..2ecd785 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -8,8 +8,9 @@ models: +materialized: table intermediate: +materialized: ephemeral - shopify__customer_email_rollup: + int_shopify__customer_email_rollup: +materialized: view # so we can use the dbt_utils.star macro + vars: shopify: shopify_customer: "{{ ref('stg_shopify__customer') }}" diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 50d3658..f18b08b 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -143,6 +143,7 @@ seeds: disjunctive: boolean published_at: timestamp updated_at: timestamp + title: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}" shopify_metafield_data: +column_types: value_type: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}" diff --git a/integration_tests/seeds/shopify_order_adjustment_data.csv b/integration_tests/seeds/shopify_order_adjustment_data.csv index 8f6edd9..191f585 100644 --- a/integration_tests/seeds/shopify_order_adjustment_data.csv +++ b/integration_tests/seeds/shopify_order_adjustment_data.csv @@ -1,6 +1,6 @@ id,order_id,refund_id,amount,tax_amount,kind,reason,amount_set,tax_amount_set,_fivetran_synced -109271056455,2712175083591,675617407047,-465,0,shipping_refund,Shipping refund,,,2020-11-14 07:52:56.522 +0000 -109277085767,2773486501959,675634708551,-95,0,shipping_refund,Shipping refund,,,2020-11-14 07:54:41.682 +0000 -109245956167,2771757826119,675548168263,-27,-1.6,shipping_refund,Shipping refund,,,2020-11-14 07:44:24.602 +0000 -109248118855,2771329908807,675555016775,-35,0,shipping_refund,Shipping refund,,,2020-11-14 07:45:11.536 +0000 -109275742279,2773429682247,675632644167,-515,0,refund_discrepancy,Refund discrepancy,,,2020-11-14 07:54:31.054 +0000 \ No newline at end of file +109271056455,2712175083591,675617407047,-465,0,shipping_refund,Shipping refund,,,2020-11-14 07:52:56.522 +109277085767,2773486501959,675634708551,-95,0,shipping_refund,Shipping refund,,,2020-11-14 07:54:41.682 +109245956167,2771757826119,675548168263,-27,-1.6,shipping_refund,Shipping refund,,,2020-11-14 07:44:24.602 +109248118855,2771329908807,675555016775,-35,0,shipping_refund,Shipping refund,,,2020-11-14 07:45:11.536 +109275742279,2773429682247,675632644167,-515,0,refund_discrepancy,Refund discrepancy,,,2020-11-14 07:54:31.054 \ No newline at end of file diff --git a/integration_tests/seeds/shopify_order_data.csv b/integration_tests/seeds/shopify_order_data.csv index 83dbb5e..fb0c0ae 100644 --- a/integration_tests/seeds/shopify_order_data.csv +++ b/integration_tests/seeds/shopify_order_data.csv @@ -1,4 +1,4 @@ id,note,email,taxes_included,currency,subtotal_price,total_tax,total_price,created_at,updated_at,name,shipping_address_name,shipping_address_first_name,shipping_address_last_name,shipping_address_company,shipping_address_phone,shipping_address_address_1,shipping_address_address_2,shipping_address_city,shipping_address_country,shipping_address_country_code,shipping_address_province,shipping_address_province_code,shipping_address_zip,shipping_address_latitude,shipping_address_longitude,billing_address_name,billing_address_first_name,billing_address_last_name,billing_address_company,billing_address_phone,billing_address_address_1,billing_address_address_2,billing_address_city,billing_address_country,billing_address_country_code,billing_address_province,billing_address_province_code,billing_address_zip,billing_address_latitude,billing_address_longitude,customer_id,location_id,user_id,number,order_number,financial_status,fulfillment_status,processed_at,processing_method,referring_site,cancel_reason,cancelled_at,closed_at,total_discounts,total_line_items_price,total_weight,source_name,browser_ip,buyer_accepts_marketing,token,cart_token,checkout_token,test,landing_site_base_url,_fivetran_synced -2674098602081,71509c29301d2cc14e37ecb53f735608,021cb20b5c78751fc7ddc091b6b69b3e,TRUE,GBP,2.8,0,2.8,2020-09-11 19:35:42.000 ,2020-09-11 19:35:46.000 ,d1743fc58a1e4d78769eaac49994a994,8b121314a4d97bc9dc15bfba8518ec88,f0962b7a185488ecb752cedac1038349,aa35cb67c26e64bb81a1bf3f17e858ba,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,d6f4a399883df85d9d4b3a02bf6e738a,bc9b8576178dcd886639ba718f1d45c8,ac08c606d455cde42980f980524a8038,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,d41d8cd98f00b204e9800998ecf8427e,,00079ce435afddc28205639142773870,d97319f64674c02595f2989019970fc8,c08dae474c5d4d3326fd6764d2a0ebe6,8b121314a4d97bc9dc15bfba8518ec88,f0962b7a185488ecb752cedac1038349,aa35cb67c26e64bb81a1bf3f17e858ba,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,d6f4a399883df85d9d4b3a02bf6e738a,bc9b8576178dcd886639ba718f1d45c8,ac08c606d455cde42980f980524a8038,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,d41d8cd98f00b204e9800998ecf8427e,,00079ce435afddc28205639142773870,d97319f64674c02595f2989019970fc8,c08dae474c5d4d3326fd6764d2a0ebe6,3589760876641,,,4135,5135,paid,,2020-09-11 19:35:42.000 ,,,,,,2.8,5.6,0,294517,,TRUE,0f9c2880de17f71511eee5542c29b999,,,FALSE,,2020-09-12 00:15:10.199 -2669516488801,,dce90c7b4e52e045e5975836aff49cf1,TRUE,GBP,2.8,0,3.79,2020-09-09 23:01:54.000 ,2020-09-10 15:38:26.000 ,4fcb884b5b46413bae526a6e7e49d706,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,3584045351009,,,4066,5066,paid,fulfilled,2020-09-09 23:01:53.000 ,direct,2cc983716a820bc713b793a6e8e73f42,,,2020-09-10 15:38:26.000 ,0,2.8,0,web,109.249.185.68,FALSE,fb489b3ccc0ae36ce47744d7595e9746,b1ff04883dfeab658cd5211050476729,7bdb994e1196de3e4f34586e357613f9,FALSE,8584e97b29b0802fb393fa453a8b6a7a,2020-09-11 00:14:33.536 -2669509541985,,dce90c7b4e52e045e5975836aff49cf1,TRUE,GBP,4.4,0,5.39,2020-09-09 22:57:51.000 ,2020-09-10 15:38:25.000 ,9e346f2e912c60e16679f4a4c8d29422,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,3584045351009,,,4065,5065,paid,fulfilled,2020-09-09 22:57:50.000 ,direct,2cc983716a820bc713b793a6e8e73f42,,,2020-09-10 15:38:25.000 ,0,4.4,0,web,109.249.185.68,FALSE,e44b7f04610a8f4032530cc7f12663de,9600543f4d4613db59ac58a1009ecbb9,cf0a9fe2c7c606b86559007dbb890a62,FALSE,8584e97b29b0802fb393fa453a8b6a7a,2020-09-11 00:14:33.037 \ No newline at end of file +2674098602081,71509c29301d2cc14e37ecb53f735608,021cb20b5c78751fc7ddc091b6b69b3e,TRUE,GBP,2.8,0,2.8,2020-09-11 19:35:42.000,2020-09-11 19:35:46.000,d1743fc58a1e4d78769eaac49994a994,8b121314a4d97bc9dc15bfba8518ec88,f0962b7a185488ecb752cedac1038349,aa35cb67c26e64bb81a1bf3f17e858ba,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,d6f4a399883df85d9d4b3a02bf6e738a,bc9b8576178dcd886639ba718f1d45c8,ac08c606d455cde42980f980524a8038,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,d41d8cd98f00b204e9800998ecf8427e,,00079ce435afddc28205639142773870,d97319f64674c02595f2989019970fc8,c08dae474c5d4d3326fd6764d2a0ebe6,8b121314a4d97bc9dc15bfba8518ec88,f0962b7a185488ecb752cedac1038349,aa35cb67c26e64bb81a1bf3f17e858ba,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,d6f4a399883df85d9d4b3a02bf6e738a,bc9b8576178dcd886639ba718f1d45c8,ac08c606d455cde42980f980524a8038,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,d41d8cd98f00b204e9800998ecf8427e,,00079ce435afddc28205639142773870,d97319f64674c02595f2989019970fc8,c08dae474c5d4d3326fd6764d2a0ebe6,3589760876641,,,4135,5135,paid,,2020-09-11 19:35:42.000,,,,,,2.8,5.6,0,294517,,TRUE,0f9c2880de17f71511eee5542c29b999,,,FALSE,,2020-09-12 00:15:10.199 +2669516488801,,dce90c7b4e52e045e5975836aff49cf1,TRUE,GBP,2.8,0,3.79,2020-09-09 23:01:54.000,2020-09-10 15:38:26.000,4fcb884b5b46413bae526a6e7e49d706,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,3584045351009,,,4066,5066,paid,fulfilled,2020-09-09 23:01:53.000,direct,2cc983716a820bc713b793a6e8e73f42,,,2020-09-10 15:38:26.000,0,2.8,0,web,109.249.185.68,FALSE,fb489b3ccc0ae36ce47744d7595e9746,b1ff04883dfeab658cd5211050476729,7bdb994e1196de3e4f34586e357613f9,FALSE,8584e97b29b0802fb393fa453a8b6a7a,2020-09-11 00:14:33.536 +2669509541985,,dce90c7b4e52e045e5975836aff49cf1,TRUE,GBP,4.4,0,5.39,2020-09-09 22:57:51.000,2020-09-10 15:38:25.000,9e346f2e912c60e16679f4a4c8d29422,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,c8189c7add9755e66391b58ecc12b3e2,d3bae70c9d49bb7cb5a74cdd0eae7fc4,0dd89cff60965dff8f9ea2bc952a5474,d41d8cd98f00b204e9800998ecf8427e,d41d8cd98f00b204e9800998ecf8427e,1ff1de774005f8da13f42943881c655f,70111f8840ccbd8b1007cc3f387ced6b,1ac412baeba98370017c73df41c98a07,89f9c9f489be2a83cf57e53b9197d288,79cba1185463850dedba31f172f1dc5b,,,2357e65b582faa0a2da3603b16fa4a7f,75c29d6dd29594a652fcbd7c4c279a29,75468fbebc28e02ec5d4f54f4cbd4099,3584045351009,,,4065,5065,paid,fulfilled,2020-09-09 22:57:50.000,direct,2cc983716a820bc713b793a6e8e73f42,,,2020-09-10 15:38:25.000,0,4.4,0,web,109.249.185.68,FALSE,e44b7f04610a8f4032530cc7f12663de,9600543f4d4613db59ac58a1009ecbb9,cf0a9fe2c7c606b86559007dbb890a62,FALSE,8584e97b29b0802fb393fa453a8b6a7a,2020-09-11 00:14:33.037 \ No newline at end of file diff --git a/integration_tests/seeds/shopify_product_variant_data.csv b/integration_tests/seeds/shopify_product_variant_data.csv index 18dcd0c..44dde0f 100644 --- a/integration_tests/seeds/shopify_product_variant_data.csv +++ b/integration_tests/seeds/shopify_product_variant_data.csv @@ -1,6 +1,6 @@ id,product_id,inventory_item_id,title,price,sku,position,inventory_policy,compare_at_price,fulfillment_service,inventory_management,created_at,updated_at,taxable,barcode,grams,image_id,inventory_quantity,weight,weight_unit,old_inventory_quantity,requires_shipping,_fivetran_synced,option_2,tax_code,option_3,option_1 -39262114414663,6540108431431,41356021661767,my title here,111,,1,deny,,manual,,2021-03-08 16:30:15.000 +0000,2021-04-12 19:49:43.000 +0000,FALSE,,0,,0,0,lb,0,FALSE,2021-04-16 07:50:32.995 +0000,,,,my title here -39273118957639,6544066379847,41367035936839,my title here,222,,1,deny,,manual,,2021-03-17 16:39:45.000 +0000,2021-04-12 19:46:59.000 +0000,FALSE,,0,,0,0,lb,0,FALSE,2021-04-16 07:50:29.241 +0000,,,,my title here -39290169262151,6548438188103,41384094924871,my title here,5,,1,deny,,manual,inventory manager,2021-03-30 19:48:15.000 +0000,2021-03-30 19:48:15.000 +0000,TRUE,,0,,0,0,lb,0,TRUE,2021-04-16 07:50:32.720 +0000,,,,my title here -39262115397703,6540109250631,41356022644807,my title here,333,,1,deny,,manual,,2021-03-08 16:31:31.000 +0000,2021-04-12 19:47:26.000 +0000,FALSE,,0,,-5,0,lb,-5,FALSE,2021-04-16 07:50:29.822 +0000,,,,my title here -29217058947142,3879735590982,30309980143686,my other title,444,,1,deny,,manual,inventory manager,2019-06-25 18:32:03.000 +0000,2019-10-01 23:40:09.000 +0000,TRUE,,222,,0,1,lb,0,TRUE,2021-04-16 07:50:25.006 +0000,,TR9999,,my other title \ No newline at end of file +39262114414663,6540108431431,41356021661767,my title here,111,,1,deny,,manual,,2021-03-08 16:30:15.000,2021-04-12 19:49:43.000,FALSE,,0,,0,0,lb,0,FALSE,2021-04-16 07:50:32.995,,,,my title here +39273118957639,6544066379847,41367035936839,my title here,222,,1,deny,,manual,,2021-03-17 16:39:45.000,2021-04-12 19:46:59.000,FALSE,,0,,0,0,lb,0,FALSE,2021-04-16 07:50:29.241,,,,my title here +39290169262151,6548438188103,41384094924871,my title here,5,,1,deny,,manual,inventory manager,2021-03-30 19:48:15.000,2021-03-30 19:48:15.000,TRUE,,0,,0,0,lb,0,TRUE,2021-04-16 07:50:32.720,,,,my title here +39262115397703,6540109250631,41356022644807,my title here,333,,1,deny,,manual,,2021-03-08 16:31:31.000,2021-04-12 19:47:26.000,FALSE,,0,,-5,0,lb,-5,FALSE,2021-04-16 07:50:29.822,,,,my title here +29217058947142,3879735590982,30309980143686,my other title,444,,1,deny,,manual,inventory manager,2019-06-25 18:32:03.000,2019-10-01 23:40:09.000,TRUE,,222,,0,1,lb,0,TRUE,2021-04-16 07:50:25.006,,TR9999,,my other title \ No newline at end of file diff --git a/integration_tests/seeds/shopify_refund_data.csv b/integration_tests/seeds/shopify_refund_data.csv index 3ed823d..7a7f50f 100644 --- a/integration_tests/seeds/shopify_refund_data.csv +++ b/integration_tests/seeds/shopify_refund_data.csv @@ -1,6 +1,6 @@ id,created_at,processed_at,note,restock,user_id,_fivetran_synced,total_duties_set,order_id -801704738887,2021-04-17 20:25:08.000 +0000,2021-04-17 20:25:08.000 +0000,,FALSE,40467791943,2021-04-18 08:05:22.056 +0000,,3726667481159 -801695039559,2021-04-17 15:45:21.000 +0000,2021-04-17 15:45:21.000 +0000,,FALSE,40467791943,2021-04-18 07:52:19.104 +0000,,3725521846343 -801704181831,2021-04-17 20:15:01.000 +0000,2021-04-17 20:15:01.000 +0000,,FALSE,40467791943,2021-04-18 08:05:22.522 +0000,,3726619476039 -801703428167,2021-04-17 19:56:51.000 +0000,2021-04-17 19:56:51.000 +0000,my refund note,FALSE,40467791943,2021-04-18 08:05:22.841 +0000,,3726370996295 -801707360327,2021-04-17 21:32:50.000 +0000,2021-04-17 21:32:50.000 +0000,,FALSE,40467791943,2021-04-18 08:02:24.256 +0000,,3726858289223 \ No newline at end of file +801704738887,2021-04-17 20:25:08.000,2021-04-17 20:25:08.000,,FALSE,40467791943,2021-04-18 08:05:22.056,,3726667481159 +801695039559,2021-04-17 15:45:21.000,2021-04-17 15:45:21.000,,FALSE,40467791943,2021-04-18 07:52:19.104,,3725521846343 +801704181831,2021-04-17 20:15:01.000,2021-04-17 20:15:01.000,,FALSE,40467791943,2021-04-18 08:05:22.522,,3726619476039 +801703428167,2021-04-17 19:56:51.000,2021-04-17 19:56:51.000,my refund note,FALSE,40467791943,2021-04-18 08:05:22.841,,3726370996295 +801707360327,2021-04-17 21:32:50.000,2021-04-17 21:32:50.000,,FALSE,40467791943,2021-04-18 08:02:24.256,,3726858289223 \ No newline at end of file diff --git a/integration_tests/seeds/shopify_transaction_data.csv b/integration_tests/seeds/shopify_transaction_data.csv index 7808885..bef96b7 100644 --- a/integration_tests/seeds/shopify_transaction_data.csv +++ b/integration_tests/seeds/shopify_transaction_data.csv @@ -1,6 +1,6 @@ id,order_id,refund_id,amount,authorization,created_at,processed_at,device_id,gateway,source_name,message,currency,location_id,parent_id,payment_avs_result_code,kind,currency_exchange_id,currency_exchange_adjustment,currency_exchange_original_amount,currency_exchange_final_amount,currency_exchange_currency,error_code,status,test,user_id,_fivetran_synced,payment_credit_card_bin,payment_cvv_result_code,payment_credit_card_number,payment_credit_card_company,receipt -2667417567303,2181743870023,,415,abcd999999,2020-02-27 16:05:37.000 +0000,2020-02-27 16:05:37.000 +0000,,gateway_here,source_name,message_here,USD,,,Z,sale,,,,,,,success,FALSE,,2020-10-28 20:33:09.797 +0000,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": null } }] }} -2572210896967,2089104834631,,415,abcd888888,2020-01-12 20:06:37.000 +0000,2020-01-12 20:06:37.000 +0000,,gateway_here,source_name,message_here,USD,,,Y,sale,,,,,,,success,FALSE,,2020-10-28 17:05:27.756 +0000,,,,, -2664325611591,2179107356743,,415,abcd77777,2020-02-26 00:12:37.000 +0000,2020-02-26 00:12:37.000 +0000,,gateway_here,source_name,message_here,USD,,,,sale,,,,,,,success,FALSE,,2020-10-28 20:23:50.344 +0000,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": "0.523" } }] }} -2595729735751,2114590769223,,15.95,abcd66666,2020-01-26 11:04:41.000 +0000,2020-01-26 11:04:41.000 +0000,,gateway_here,source_name,message_here,USD,,,Y,sale,,,,,,,success,FALSE,,2020-10-28 18:10:27.604 +0000,,,,, -2705030512711,2214516916295,,212.12,abcd5555,2020-03-18 00:17:24.000 +0000,2020-03-18 00:17:24.000 +0000,,gateway_here,source_name,message_here,USD,,,,sale,,,,,,,success,FALSE,,2020-10-28 22:14:02.944 +0000,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": "0.96581" } }] }} \ No newline at end of file +2667417567303,2181743870023,,415,abcd999999,2020-02-27 16:05:37.000,2020-02-27 16:05:37.000,,gateway_here,source_name,message_here,USD,,,Z,sale,,,,,,,success,FALSE,,2020-10-28 20:33:09.797,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": null } }] }} +2572210896967,2089104834631,,415,abcd888888,2020-01-12 20:06:37.000,2020-01-12 20:06:37.000,,gateway_here,source_name,message_here,USD,,,Y,sale,,,,,,,success,FALSE,,2020-10-28 17:05:27.756,,,,, +2664325611591,2179107356743,,415,abcd77777,2020-02-26 00:12:37.000,2020-02-26 00:12:37.000,,gateway_here,source_name,message_here,USD,,,,sale,,,,,,,success,FALSE,,2020-10-28 20:23:50.344,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": "0.523" } }] }} +2595729735751,2114590769223,,15.95,abcd66666,2020-01-26 11:04:41.000,2020-01-26 11:04:41.000,,gateway_here,source_name,message_here,USD,,,Y,sale,,,,,,,success,FALSE,,2020-10-28 18:10:27.604,,,,, +2705030512711,2214516916295,,212.12,abcd5555,2020-03-18 00:17:24.000,2020-03-18 00:17:24.000,,gateway_here,source_name,message_here,USD,,,,sale,,,,,,,success,FALSE,,2020-10-28 22:14:02.944,,,,,{ "charges": { "data": [ { "balance_transaction": { "exchange_rate": "0.96581" } }] }} \ No newline at end of file diff --git a/models/intermediate/shopify__customer_email_rollup.sql b/models/intermediate/int_shopify__customer_email_rollup.sql similarity index 100% rename from models/intermediate/shopify__customer_email_rollup.sql rename to models/intermediate/int_shopify__customer_email_rollup.sql diff --git a/models/intermediate/shopify__emails__order_aggregates.sql b/models/intermediate/int_shopify__emails__order_aggregates.sql similarity index 100% rename from models/intermediate/shopify__emails__order_aggregates.sql rename to models/intermediate/int_shopify__emails__order_aggregates.sql diff --git a/models/intermediate/int_shopify__inventory_level__aggregates.sql b/models/intermediate/int_shopify__inventory_level__aggregates.sql new file mode 100644 index 0000000..f879ffc --- /dev/null +++ b/models/intermediate/int_shopify__inventory_level__aggregates.sql @@ -0,0 +1,94 @@ +with order_lines as ( + + select * + from {{ var('shopify_order_line') }} +), + +fulfillment as ( + + select * + from {{ var('shopify_fulfillment') }} +), + +orders as ( + + select * + from {{ var('shopify_order') }} + where not coalesce(is_deleted, false) +), + +refunds as ( + + select * + from {{ ref('shopify__orders__order_refunds') }} + +), refunds_aggregated as ( + + select + order_line_id, + source_relation, + sum(quantity) as quantity, + sum(coalesce(subtotal, 0)) as subtotal + + from refunds + group by 1,2 +), + +joined as ( + + select + order_lines.order_line_id, + order_lines.variant_id, + order_lines.source_relation, + fulfillment.location_id, -- location id is stored in fulfillment rather than order + orders.order_id, + orders.customer_id, + lower(orders.email) as email, + order_lines.pre_tax_price, + order_lines.quantity, + orders.created_timestamp as order_created_timestamp, + fulfillment.status as fulfillment_status, + refunds_aggregated.subtotal as subtotal_sold_refunds, + refunds_aggregated.quantity as quantity_sold_refunds + + from order_lines + join orders + on order_lines.order_id = orders.order_id + and order_lines.source_relation = orders.source_relation + join fulfillment + on orders.order_id = fulfillment.order_id + and orders.source_relation = fulfillment.source_relation + + left join refunds_aggregated + on refunds_aggregated.order_line_id = order_lines.order_line_id + and refunds_aggregated.source_relation = order_lines.source_relation +), + +aggregated as ( + + select + variant_id, + location_id, + source_relation, + sum(pre_tax_price) as subtotal_sold, + sum(quantity) as quantity_sold, + count(distinct order_id) as count_distinct_orders, + count(distinct customer_id) as count_distinct_customers, + count(distinct email) as count_distinct_customer_emails, + min(order_created_timestamp) as first_order_timestamp, + max(order_created_timestamp) as last_order_timestamp + + {% for status in ['pending', 'open', 'success', 'cancelled', 'error', 'failure'] %} + , sum(case when fulfillment_status = '{{ status }}' then 1 else 0 end) as count_fulfillment_{{ status }} + {% endfor %} + + , sum(coalesce(subtotal_sold_refunds, 0)) as subtotal_sold_refunds + , sum(coalesce(quantity_sold_refunds, 0)) as quantity_sold_refunds + + from joined + + {{ dbt_utils.group_by(n=3) }} +) + +select * +from aggregated \ No newline at end of file diff --git a/models/intermediate/int_shopify__products_with_aggregates.sql b/models/intermediate/int_shopify__products_with_aggregates.sql new file mode 100644 index 0000000..c4da9d2 --- /dev/null +++ b/models/intermediate/int_shopify__products_with_aggregates.sql @@ -0,0 +1,110 @@ +with products as ( + + select * + from {{ var('shopify_product') }} +), + +collection_product as ( + + select * + from {{ var('shopify_collection_product') }} +), + +collection as ( + + select * + from {{ var('shopify_collection') }} + where not coalesce(is_deleted, false) -- limit to only active collections +), + +product_tag as ( + + select * + from {{ var('shopify_product_tag') }} +), + +product_variant as ( + + select * + from {{ var('shopify_product_variant') }} +), + +product_image as ( + + select * + from {{ var('shopify_product_image') }} +), + + +collections_aggregated as ( + + select + collection_product.product_id, + collection_product.source_relation, + {{ fivetran_utils.string_agg(field_to_agg='collection.title', delimiter="', '") }} as collections + from collection_product + join collection + on collection_product.collection_id = collection.collection_id + and collection_product.source_relation = collection.source_relation + group by 1,2 +), + +tags_aggregated as ( + + select + product_id, + source_relation, + {{ fivetran_utils.string_agg(field_to_agg='value', delimiter="', '") }} as tags + + from product_tag + group by 1,2 +), + +variants_aggregated as ( + + select + product_id, + source_relation, + count(variant_id) as count_variants + + from product_variant + group by 1,2 + +), + +images_aggregated as ( + + select + product_id, + source_relation, + count(*) as count_images + from product_image + group by 1,2 +), + +joined as ( + + select + products.*, + collections_aggregated.collections, + tags_aggregated.tags, + variants_aggregated.count_variants, + coalesce(images_aggregated.count_images, 0) > 0 as has_product_image + + from products + left join collections_aggregated + on products.product_id = collections_aggregated.product_id + and products.source_relation = collections_aggregated.source_relation + left join tags_aggregated + on products.product_id = tags_aggregated.product_id + and products.source_relation = tags_aggregated.source_relation + left join variants_aggregated + on products.product_id = variants_aggregated.product_id + and products.source_relation = variants_aggregated.source_relation + left join images_aggregated + on products.product_id = images_aggregated.product_id + and products.source_relation = images_aggregated.source_relation +) + +select * +from joined \ No newline at end of file diff --git a/models/intermediate/intermediate.yml b/models/intermediate/intermediate.yml index 9c5decb..401b3b5 100644 --- a/models/intermediate/intermediate.yml +++ b/models/intermediate/intermediate.yml @@ -14,15 +14,22 @@ models: - order_id - source_relation - name: shopify__orders__order_refunds - - name: shopify__emails__order_aggregates + - name: int_shopify__emails__order_aggregates tests: - dbt_utils.unique_combination_of_columns: combination_of_columns: - email - source_relation - - name: shopify__customer_email_rollup + - name: int_shopify__customer_email_rollup tests: - dbt_utils.unique_combination_of_columns: combination_of_columns: - email + - source_relation + - name: int_hopify__inventory_level__aggregates + tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - variant_id + - location_id - source_relation \ No newline at end of file diff --git a/models/intermediate/shopify__orders__order_refunds.sql b/models/intermediate/shopify__orders__order_refunds.sql index d91e387..cd775ee 100644 --- a/models/intermediate/shopify__orders__order_refunds.sql +++ b/models/intermediate/shopify__orders__order_refunds.sql @@ -1,5 +1,3 @@ -{{ config(enabled=fivetran_utils.enabled_vars(['shopify__using_order_line_refund','shopify__using_refund'])) }} - with refunds as ( select * diff --git a/models/shopify.yml b/models/shopify.yml index 59bebef..11e77ac 100644 --- a/models/shopify.yml +++ b/models/shopify.yml @@ -555,3 +555,170 @@ models: description: The exchange rate between the home currency and the currency of sale at the time of the transaction. - name: currency_exchange_calculated_amount description: The total amount of the transaction with the currency exchange rate applied. + + - name: shopify__inventory_levels + description: > + Model representing the current status of an inventory level in Shopify. Inventory Levels are unique pairings of + inventory items (which have a 1:1 relationship with product variants) and locations. + + Inventory Level = Inventory item @ a Location. + tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - inventory_item_id + - location_id + - source_relation + columns: + - name: _fivetran_synced + description: "{{ doc('_fivetran_synced') }}" + - name: available_quantity + description: The available quantity of an inventory item at the inventory level's associated location. Returns null if the inventory item is not tracked. + - name: inventory_item_id + description: The ID of the inventory item associated with the inventory level. + tests: + - not_null + - name: location_id + description: The ID of the location that the inventory level belongs to. + tests: + - not_null + - name: updated_at + description: The date and time (ISO 8601 format) when the inventory level was last modified. + - name: is_inventory_item_deleted + description: "{{ doc('_fivetran_deleted') }}" + - name: sku + description: The unique SKU (stock keeping unit) of the inventory item. + - name: cost + description: The unit cost of the inventory item. The shop's default currency is used. + - name: province_code_of_origin + description: The province code (ISO 3166-2 alpha-2) of where the item came from. The province code is only used if the shipping provider for the inventory item is Canada Post. + - name: country_code_of_origin + description: The country code (ISO 3166-1 alpha-2) of where the item came from. + - name: is_shipping_required + description: Boolean representing whether a customer needs to provide a shipping address when placing an order containing the inventory item. + - name: is_inventory_quantity_tracked + description: Boolean representing whether inventory levels are tracked for the item. If true, then the inventory quantity changes are tracked by Shopify. + - name: inventory_item_created_at + description: The date and time (ISO 8601 format) when the inventory item was created. + - name: inventory_item_updated_at + description: The date and time (ISO 8601 format) when the inventory item was last modified. + - name: is_location_deleted + description: "{{ doc('_fivetran_deleted') }}" + - name: is_location_active + description: > + Boolean representing whether the location is active. If true, then the location can be used to sell products, stock inventory, and fulfill orders. + - name: address_1 + description: The location's street address. + - name: address_2 + description: The optional second line of the location's street address. + - name: city + description: The city the location is in. + - name: country + description: The country the location is in (two-letter code). + - name: country_code + description: The two-letter code (ISO 3166-1 alpha-2 format) corresponding to country the location is in. + - name: location_created_at + description: The date and time (ISO 8601 format) when the location was created. + - name: is_legacy_location + description: > + Boolean representing whether this is a fulfillment service location. If true, then the location is a fulfillment service location. + If false, then the location was created by the merchant and isn't tied to a fulfillment service. + - name: location_name + description: The name of the location. + - name: phone + description: The phone number of the location. This value can contain special characters, such as - or +. + - name: province + description: The province, state, or district of the location. + - name: province_code + description: The province, state, or district code (ISO 3166-2 alpha-2 format) of the location. + - name: location_updated_at + description: The date and time (ISO 8601 format) when the location was last updated. + - name: zip + description: The zip or postal code. + - name: variant_barcode + description: The barcode, UPC, or ISBN number for the product. + - name: variant_created_at + description: The date and time (ISO 8601 format) when the product variant was created. + - name: variant_fulfillment_service + description: The fulfillment service associated with the product variant. + - name: variant_grams + description: The weight of the product variant in grams. + - name: variant_id + description: The unique numeric identifier for the product variant. + - name: variant_image_id + description: The unique numeric identifier for a product's image. The image must be associated to the same product as the variant. + - name: variant_inventory_management + description: The fulfillment service that tracks the number of items in stock for the product variant. + - name: variant_inventory_policy + description: Whether customers are allowed to place an order for the product variant when it's out of stock. + - name: variant_inventory_quantity + description: An aggregate of inventory across all locations. To adjust inventory at a specific location, use the InventoryLevel resource. + - name: variant_option_1 + description: "The custom properties that a shop owner uses to define product variants. You can define three options for a product variant: option1, option2, option3." + - name: variant_option_2 + description: "The custom properties that a shop owner uses to define product variants. You can define three options for a product variant: option1, option2, option3." + - name: variant_option_3 + description: "The custom properties that a shop owner uses to define product variants. You can define three options for a product variant: option1, option2, option3." + - name: variant_price + description: The price of the product variant. + - name: product_id + description: The unique numeric identifier for the product. + - name: is_variant_taxable + description: Whether a tax is charged when the product variant is sold. + - name: variant_tax_code + description: This parameter applies only to the stores that have the Avalara AvaTax app installed. Specifies the Avalara tax code for the product variant. + - name: variant_title + description: The title of the product variant. The title field is a concatenation of the option1, option2, and option3 fields. You can only update title indirectly using the option fields. + - name: variant_updated_at + description: The date and time when the product variant was last modified. Gets returned in ISO 8601 format. + - name: variant_weight + description: The weight of the product variant in the unit system specified with weight_unit. + - name: variant_weight_unit + description: "The unit of measurement that applies to the product variant's weight. If you don't specify a value for weight_unit, then the shop's default unit of measurement is applied. Valid values: g, kg, oz, and lb." + - name: subtotal_sold + description: Total amount (monetary) sold from the inventory level. Includes refunds. + - name: quantity_sold + description: Total quantity sold from the inventory level. Includes refunds. + - name: count_distinct_orders + description: The number of distinct orders placed from this inventory level. + - name: count_distinct_customers + description: The number of distinct customers (based on customer_id) that have placed orders against this inventory level. + - name: count_distinct_customer_emails + description: The number of distinct customer emails that have placed orders against this inventory level. + - name: first_order_timestamp + description: The timetamp of the first order against this inventory level. + - name: last_order_timestamp + description: The timetamp of the first order against this inventory level. + - name: subtotal_sold_refunds + description: The monetary amount of inventory level goods that have been refunded. + - name: quantity_sold_refunds + description: The quantity of inventory level goods that have been refunded. + - name: net_subtotal_sold + description: Net monetary amount sold from the inventory level. Excludes refunds. + - name: net_quantity_sold + description: Net quantity sold from this inventory levle. Excludes refunds. + - name: count_fulfillment_pending + description: > + Count of currently pending fulfillments for this inventory level. + "Pending" = Shopify has created the fulfillment and is waiting for the third-party fulfillment service to transition it to 'open' or 'success'. + - name: count_fulfillment_open + description: > + Count of currently open fulfillments for this inventory level. + "Open" = The fulfillment has been acknowledged by the service and is in processing. + - name: count_fulfillment_success + description: > + Count of successful fulfillments for this inventory level. + "Success" = The fulfillment was successful. + - name: count_fulfillment_cancelled + description: > + Count of cancelled fulfillments for this inventory level. + "Cancelled" = The fulfillment was cancelled. + - name: count_fulfillment_error + description: > + Count of fulfillments for this inventory level that encountered an error. + "Error" = There was an error with the fulfillment request. + - name: count_fulfillment_failure + description: > + Count of fulfillments for this inventory level that failed. + "Failure" = The fulfillment request failed. + - name: source_relation + description: The schema or database this record came from, if you are unioning multiple connectors. Null if not. \ No newline at end of file diff --git a/models/shopify__customer_emails.sql b/models/shopify__customer_emails.sql index 1e9be41..1b70210 100644 --- a/models/shopify__customer_emails.sql +++ b/models/shopify__customer_emails.sql @@ -1,13 +1,13 @@ with customer_emails as ( select - {{ dbt_utils.star(from=ref('shopify__customer_email_rollup'), except=["orders_count", "total_spent"]) }} - from {{ ref('shopify__customer_email_rollup') }} + {{ dbt_utils.star(from=ref('int_shopify__customer_email_rollup'), except=["orders_count", "total_spent"]) }} + from {{ ref('int_shopify__customer_email_rollup') }} ), orders as ( select * - from {{ ref('shopify__emails__order_aggregates' )}} + from {{ ref('int_shopify__emails__order_aggregates' )}} ), joined as ( diff --git a/models/shopify__inventory_levels.sql b/models/shopify__inventory_levels.sql new file mode 100644 index 0000000..489e4ec --- /dev/null +++ b/models/shopify__inventory_levels.sql @@ -0,0 +1,138 @@ +with inventory_level as ( + + select * + from {{ var('shopify_inventory_level') }} +), + +inventory_item as ( + + select * + from {{ var('shopify_inventory_item') }} +), + +location as ( + + select * + from {{ var('shopify_location') }} +), + +product_variant as ( + + select * + from {{ var('shopify_product_variant') }} +), + +product as ( + + select * + from {{ var('shopify_product') }} +), + +inventory_level_aggregated as ( + + select * + from {{ ref('int_shopify__inventory_level__aggregates') }} +), + +joined_info as ( + + select + inventory_level.*, + inventory_item.sku, + inventory_item.is_deleted as is_inventory_item_deleted, + inventory_item.cost, + inventory_item.country_code_of_origin, + inventory_item.province_code_of_origin, + inventory_item.is_shipping_required, + inventory_item.is_inventory_quantity_tracked, + inventory_item.created_at as inventory_item_created_at, + inventory_item.updated_at as inventory_item_updated_at, + + location.name as location_name, + location.is_deleted as is_location_deleted, + location.is_active as is_location_active, + location.address_1, + location.address_2, + location.city, + location.country, + location.country_code, + location.is_legacy as is_legacy_location, + location.province, + location.province_code, + location.phone, + location.zip, + location.created_at as location_created_at, + location.updated_at as location_updated_at, + + product_variant.variant_id, + product_variant.product_id, + product_variant.title as variant_title, + product_variant.inventory_policy as variant_inventory_policy, + product_variant.price as variant_price, + product_variant.image_id as variant_image_id, + product_variant.fulfillment_service as variant_fulfillment_service, + product_variant.inventory_management as variant_inventory_management, + product_variant.is_taxable as is_variant_taxable, + product_variant.barcode as variant_barcode, + product_variant.grams as variant_grams, + product_variant.inventory_quantity as variant_inventory_quantity, + product_variant.weight as variant_weight, + product_variant.weight_unit as variant_weight_unit, + product_variant.option_1 as variant_option_1, + product_variant.option_2 as variant_option_2, + product_variant.option_3 as variant_option_3, + product_variant.tax_code as variant_tax_code, + product_variant.created_timestamp as variant_created_at, + product_variant.updated_timestamp as variant_updated_at + + {{ fivetran_utils.persist_pass_through_columns('product_variant_pass_through_columns', identifier='product_variant') }} + + from inventory_level + join inventory_item + on inventory_level.inventory_item_id = inventory_item.inventory_item_id + and inventory_level.source_relation = inventory_item.source_relation + join location + on inventory_level.location_id = location.location_id + and inventory_level.source_relation = location.source_relation + join product_variant + on inventory_item.inventory_item_id = product_variant.inventory_item_id + and inventory_item.source_relation = product_variant.source_relation + +), + +joined_aggregates as ( + + select + joined_info.*, + coalesce(inventory_level_aggregated.subtotal_sold, 0) as subtotal_sold, + coalesce(inventory_level_aggregated.quantity_sold, 0) as quantity_sold, + coalesce(inventory_level_aggregated.count_distinct_orders, 0) as count_distinct_orders, + coalesce(inventory_level_aggregated.count_distinct_customers, 0) as count_distinct_customers, + coalesce(inventory_level_aggregated.count_distinct_customer_emails, 0) as count_distinct_customer_emails, + inventory_level_aggregated.first_order_timestamp, + inventory_level_aggregated.last_order_timestamp, + coalesce(inventory_level_aggregated.subtotal_sold_refunds, 0) as subtotal_sold_refunds, + coalesce(inventory_level_aggregated.quantity_sold_refunds, 0) as quantity_sold_refunds + + {% for status in ['pending', 'open', 'success', 'cancelled', 'error', 'failure'] %} + , coalesce(count_fulfillment_{{ status }}, 0) as count_fulfillment_{{ status }} + {% endfor %} + + from joined_info + left join inventory_level_aggregated + on joined_info.location_id = inventory_level_aggregated.location_id + and joined_info.variant_id = inventory_level_aggregated.variant_id + and joined_info.source_relation = inventory_level_aggregated.source_relation +), + +final as ( + + select + *, + subtotal_sold - subtotal_sold_refunds as net_subtotal_sold, + quantity_sold - quantity_sold_refunds as net_quantity_sold + from joined_aggregates +) + +select * +from final \ No newline at end of file diff --git a/models/shopify__order_lines.sql b/models/shopify__order_lines.sql index 6fe4f71..1b1f8de 100644 --- a/models/shopify__order_lines.sql +++ b/models/shopify__order_lines.sql @@ -8,7 +8,6 @@ with order_lines as ( select * from {{ var('shopify_product_variant') }} -{% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} ), refunds as ( select * @@ -23,32 +22,29 @@ with order_lines as ( sum(coalesce(subtotal, 0)) as subtotal from refunds group by 1,2 -{% endif %} ), joined as ( select order_lines.*, - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} coalesce(refunds_aggregated.quantity,0) as refunded_quantity, coalesce(refunds_aggregated.subtotal,0) as refunded_subtotal, order_lines.quantity - coalesce(refunds_aggregated.quantity,0) as quantity_net_refunds, order_lines.pre_tax_price - coalesce(refunds_aggregated.subtotal,0) as subtotal_net_refunds, - {% endif %} product_variants.created_timestamp as variant_created_at, product_variants.updated_timestamp as variant_updated_at, product_variants.inventory_item_id, product_variants.image_id, - product_variants.title as variant_title, + product_variants.price as variant_price, product_variants.sku as variant_sku, product_variants.position as variant_position, product_variants.inventory_policy as variant_inventory_policy, product_variants.compare_at_price as variant_compare_at_price, product_variants.fulfillment_service as variant_fulfillment_service, - product_variants.inventory_management as variant_inventory_management, + product_variants.is_taxable as variant_is_taxable, product_variants.barcode as variant_barcode, product_variants.grams as variant_grams, @@ -60,14 +56,12 @@ with order_lines as ( product_variants.option_3 as variant_option_3, product_variants.tax_code as variant_tax_code {# , - use inventoryitem or order line itself + use inventoryitem or order line itself -- add later TODO product_variants.is_requiring_shipping as variant_is_requiring_shipping #} from order_lines - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} left join refunds_aggregated on refunds_aggregated.order_line_id = order_lines.order_line_id and refunds_aggregated.source_relation = order_lines.source_relation - {% endif %} left join product_variants on product_variants.variant_id = order_lines.variant_id and product_variants.source_relation = order_lines.source_relation diff --git a/models/shopify__orders.sql b/models/shopify__orders.sql index 8c3684c..599e31c 100644 --- a/models/shopify__orders.sql +++ b/models/shopify__orders.sql @@ -8,7 +8,6 @@ with orders as ( select * from {{ ref('shopify__orders__order_line_aggregates') }} -{% if var('shopify__using_order_adjustment', true) %} ), order_adjustments as ( select * @@ -22,9 +21,7 @@ with orders as ( sum(tax_amount) as order_adjustment_tax_amount from order_adjustments group by 1,2 -{% endif %} -{% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} ), refunds as ( select * @@ -38,7 +35,6 @@ with orders as ( sum(total_tax) as refund_total_tax from refunds group by 1,2 -{% endif %} ), joined as ( @@ -46,38 +42,27 @@ with orders as ( orders.*, coalesce(cast({{ fivetran_utils.json_parse("total_shipping_price_set",["shop_money","amount"]) }} as {{ dbt.type_float() }}) ,0) as shipping_cost, - {% if var('shopify__using_order_adjustment', true) %} order_adjustments_aggregates.order_adjustment_amount, order_adjustments_aggregates.order_adjustment_tax_amount, - {% endif %} - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} refund_aggregates.refund_subtotal, refund_aggregates.refund_total_tax, - {% endif %} + (orders.total_price - {% if var('shopify__using_order_adjustment', true) %} + coalesce(order_adjustments_aggregates.order_adjustment_amount,0) + coalesce(order_adjustments_aggregates.order_adjustment_tax_amount,0) - {% endif %} - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} - - coalesce(refund_aggregates.refund_subtotal,0) - coalesce(refund_aggregates.refund_total_tax,0) - {% endif %} ) as order_adjusted_total, + - coalesce(refund_aggregates.refund_subtotal,0) - coalesce(refund_aggregates.refund_total_tax,0)) as order_adjusted_total, order_lines.line_item_count from orders left join order_lines on orders.order_id = order_lines.order_id and orders.source_relation = order_lines.source_relation - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} left join refund_aggregates on orders.order_id = refund_aggregates.order_id and orders.source_relation = refund_aggregates.source_relation - {% endif %} - {% if var('shopify__using_order_adjustment', true) %} left join order_adjustments_aggregates on orders.order_id = order_adjustments_aggregates.order_id and orders.source_relation = order_adjustments_aggregates.source_relation - {% endif %} ), windows as ( diff --git a/models/shopify__products.sql b/models/shopify__products.sql index 9f63dcd..21a8c81 100644 --- a/models/shopify__products.sql +++ b/models/shopify__products.sql @@ -1,19 +1,22 @@ with products as ( select * - from {{ var('shopify_product') }} + from {{ ref('int_shopify__products_with_aggregates') }} +), -), order_lines as ( +order_lines as ( select * from {{ ref('shopify__order_lines') }} +), -), orders as ( +orders as ( select * from {{ ref('shopify__orders')}} +), -), order_lines_aggregated as ( +order_lines_aggregated as ( select order_lines.product_id, @@ -21,10 +24,8 @@ with products as ( sum(order_lines.quantity) as quantity_sold, sum(order_lines.pre_tax_price) as subtotal_sold, - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} sum(order_lines.quantity_net_refunds) as quantity_sold_net_refunds, sum(order_lines.subtotal_net_refunds) as subtotal_sold_net_refunds, - {% endif %} min(orders.created_timestamp) as first_order_timestamp, max(orders.created_timestamp) as most_recent_order_timestamp @@ -33,24 +34,23 @@ with products as ( using (order_id, source_relation) group by 1,2 -), joined as ( +), + +joined as ( select products.*, coalesce(order_lines_aggregated.quantity_sold,0) as quantity_sold, coalesce(order_lines_aggregated.subtotal_sold,0) as subtotal_sold, - - {% if fivetran_utils.enabled_vars(vars=["shopify__using_order_line_refund", "shopify__using_refund"]) %} coalesce(order_lines_aggregated.quantity_sold_net_refunds,0) as quantity_sold_net_refunds, coalesce(order_lines_aggregated.subtotal_sold_net_refunds,0) as subtotal_sold_net_refunds, - {% endif %} - order_lines_aggregated.first_order_timestamp, order_lines_aggregated.most_recent_order_timestamp + from products left join order_lines_aggregated - using (product_id, source_relation) - + on products.product_id = order_lines_aggregated.product_id + and products.source_relation = order_lines_aggregated.source_relation ) select *