Adapts billing history to invoice grain

CHANGELOG.md

@@ -1,3 +1,30 @@
+# dbt_zuora v0.2.0
+[PR #8](https://github.com/fivetran/dbt_zuora/pull/8) includes the following breaking changes:
+
+## 🚨 Breaking Changes 🚨
+- Redesigned `zuora__billing_history` so `invoice_id` was the actual grain, accounting for the cases when invoices can be tied to multiple credit balance adjustments and payments. This required the following steps:
+    - All metrics associated with `credit_balance_adjustment_id` in `int_zuora__billing_enriched` were aggregated on the invoice level (number of credit balance adjustments,total credit balance amount in home currency, first and last dates of credit balance adjustments on the invoice). Fields associated with an individual `credit_balance_adjustment_id` were removed.
+    - All metrics associated with `payment_id` in `int_zuora__billing_enriched` were aggregated on the invoice level (number of payments, total payment amount in home currency, first and last dates of payments on the invoice). Fields associated with an individual `payment_id` were removed.
+    - Aggregated count of `payment_method_id` to gather a count of payment methods on each invoice.
+    - A full list of field changes in `zuora__billing_history` is provided below: 
+
+| **[Zuora__billing_history](https://fivetran.github.io/dbt_zuora/#!/model/model.zuora.zuora__billing_history)** updates | **Field changes** |
+| ---------- | -------------------- |
+| Added payment fields | `payments`, `first_payment_date`, `most_recent_payment_date`, `payment_methods` |
+| Modified payment fields  (aggregated line items) | `invoice_amount_unpaid_home_currency` |
+| Removed payment fields |  `payment_id`, `payment_number`, `payment_date`, `payment_status`, `payment_type` |
+| Added credit balance adjustment fields | `credit_balance_adjustments`, `first_credit_balance_adjustment_date`, `most_recent_credit_balance_adjustment_date` |
+| Modified credit balance adjustment fields (aggregated line items) | `credit_balance_adjustment_amount_home_currency` |
+| Removed credit balance adjustment fields | `credit_balance_adjustment_id`, `credit_balance_adjustment_number`, `credit_balance_adjustment_reason_code`, `credit_balance_adjustment_date` |
+| Added payment method fields | `payment_methods` |
+| Removed payment method fields | `payment_method_id`, `payment_method_type`, `payment_method_subtype`, `is_payment_method_active` | 
+
+- Updated yml of `zuora__billing_history` with aggregated fields, and removed non-aggregated fields associated with credit balance adjustments, payments, and payment methods.
+- Modified `int_zuora__account_enriched` to account for new aggregated metric fields being pulled from `zuora__billing_history`. 
+
+## 🔧 Under The Hood 🔩
+- Updated seed files in `integration_tests` to reproduce the initial `invoice_id` uniqueness test issue for multiple credit balance adjustments and payments for one invoice.
+
 # dbt_zuora v0.1.0
 🎉 This is the initial release of this package! 🎉
 ## 📣 What does this dbt package do?

README.md

@@ -60,7 +60,7 @@ Include the following zuora_source package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/zuora
-    version: [">=0.1.0", "<0.2.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do NOT include the `zuora_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 
@@ -179,7 +179,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/zuora_source
-      version: [">=0.1.0", "<0.2.0"]
+      version: [">=0.2.0", "<0.3.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'zuora'
-version: '0.1.0'
+version: '0.2.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'zuora_integration_tests'
-version: '0.1.0'
+version: '0.2.0'
 profile: 'integration_tests'
 config-version: 2
 

integration_tests/seeds/zuora_credit_balance_adjustment_data.csv

@@ -1,6 +1,7 @@
 id,_fivetran_deleted,_fivetran_synced,account_id,account_receivable_accounting_code_id,accounting_code,accounting_period_id,adjustment_date,amount,amount_currency_rounding,amount_home_currency,bill_to_contact_id,cancelled_on,comment,created_by_id,created_date,customer_cash_on_account_accounting_code_id,default_payment_method_id,exchange_rate,exchange_rate_date,home_currency,invoice_id,journal_entry_id,journal_run_id,number,parent_account_id,provider_exchange_rate_date,reason_code,reference_id,sold_to_contact_id,source_transaction_id,source_transaction_number,source_transaction_type,status,transaction_currency,transferred_to_accounting,type,updated_by_id,updated_date
-dff38ac0f147ac59a2d058bb31738860,FALSE,2023-02-20 23:03:01.186000,dff38ac0df6edbfb4cb3774f7e21f752,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,92914.02,,,dff38ac0a944caafecb14de7bafd756b,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:12:41.000000,dff38ac088918093b68eeac3173ad31c,dff38ac030ec94a9a491c6a5e4eaa2be,,2016-01-15,USD,dff38ac0d5148e18c0ad1db08e7af528,dff38ac04fad17667af419b63877d4aa,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000001,,2016-01-14,Standard Adjustment,,dff38ac0a944caafecb14de7bafd756b,dff38ac0d5148e18c0ad1db08e7af528,INV00000058,Invoice,Processed,AUD,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:12:41.000000
-dff38ac09e98819c968081aee810d064,FALSE,2023-02-20 23:03:01.186000,dff38ac0e5e1b210b55e785af668c300,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,85794.28,,,dff38ac0fd532ffb3826262a658a6fe8,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:02.000000,dff38ac088918093b68eeac3173ad31c,dff38ac029b398b9e5bb9ab74146ed1e,,2016-01-15,USD,dff38ac0fe708f09924dafc2dba6bdc9,dff38ac059b7dd5e83ff0167914620e7,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000002,dff38ac0e09e5a914e7cddb66f92e2af,2016-01-14,Standard Adjustment,,dff38ac0fd532ffb3826262a658a6fe8,dff38ac0fe708f09924dafc2dba6bdc9,INV00000059,Invoice,Processed,EUR,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:02.000000
-dff38ac013d167917d82b246268e5b83,FALSE,2023-02-20 23:03:01.188000,dff38ac0c42164706d15f0e7fee92ab2,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,103376.5,,,dff38ac06e013871256e292f2584dc34,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:46.000000,dff38ac088918093b68eeac3173ad31c,dff38ac0aa72639615964e432c251cf9,,2016-01-15,USD,dff38ac01e7be6b9ccabb275d0c0d355,dff38ac059b7dd5e83ff0167914620e7,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000005,,2016-01-14,Standard Adjustment,,dff38ac06e013871256e292f2584dc34,dff38ac01e7be6b9ccabb275d0c0d355,INV00000062,Invoice,Processed,EUR,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:46.000000
-dff38ac0bf5884961339694eaef29030,FALSE,2023-02-20 23:03:01.187000,dff38ac075b0b9737438a32e304763fe,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,11212.9,,,dff38ac0402fac05f894822c7262d5fe,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:33.000000,dff38ac088918093b68eeac3173ad31c,dff38ac089b849bd637f8280de505751,,2016-01-15,USD,dff38ac0e5a5c1258835301ece425def,dff38ac05d3da6c9e3d7df3eec93ed0c,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000004,,2016-01-14,Standard Adjustment,,dff38ac0402fac05f894822c7262d5fe,dff38ac0e5a5c1258835301ece425def,INV00000061,Invoice,Processed,GBP,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:33.000000
-dff38ac015cf0d852d3dfff06e63a038,FALSE,2023-02-20 23:03:01.187000,dff38ac05ae066e479f6ba9b68a23d47,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,3130859,,,dff38ac07e977c34b06543bd339ddbab,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000,dff38ac088918093b68eeac3173ad31c,dff38ac0359bb2514a0791d861546823,,2016-01-15,USD,dff38ac0bc55aae22a3d521c8c184d44,dff38ac08d0d6254630cc88ff7301563,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000003,,2016-01-14,Standard Adjustment,,dff38ac07e977c34b06543bd339ddbab,dff38ac0bc55aae22a3d521c8c184d44,INV00000060,Invoice,Processed,JPY,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000
+dff38ac0f147ac59a2d058bb31738860,FALSE,2023-02-20 23:03:01.186000,dff38ac0df6edbfb4cb3774f7e21f752,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,92914.02,,,dff38ac0a944caafecb14de7bafd756b,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:12:41.000000,dff38ac088918093b68eeac3173ad31c,dff38ac030ec94a9a491c6a5e4eaa2be,,2016-01-15,USD,dff38ac0a531a01100012160a3061ef6,dff38ac04fad17667af419b63877d4aa,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000001,,2016-01-14,Standard Adjustment,,dff38ac0a944caafecb14de7bafd756b,dff38ac0d5148e18c0ad1db08e7af528,INV00000058,Invoice,Processed,AUD,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:12:41.000000
+dff38ac09e98819c968081aee810d064,FALSE,2023-02-20 23:03:01.186000,dff38ac0e5e1b210b55e785af668c300,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,85794.28,,,dff38ac0fd532ffb3826262a658a6fe8,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:02.000000,dff38ac088918093b68eeac3173ad31c,dff38ac029b398b9e5bb9ab74146ed1e,,2016-01-15,USD,dff38ac0465fa8828673ba7d616b6ea1,dff38ac059b7dd5e83ff0167914620e7,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000002,dff38ac0e09e5a914e7cddb66f92e2af,2016-01-14,Standard Adjustment,,dff38ac0fd532ffb3826262a658a6fe8,dff38ac0fe708f09924dafc2dba6bdc9,INV00000059,Invoice,Processed,EUR,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:02.000000
+dff38ac013d167917d82b246268e5b83,FALSE,2023-02-20 23:03:01.188000,dff38ac0c42164706d15f0e7fee92ab2,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,103376.5,,,dff38ac06e013871256e292f2584dc34,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:46.000000,dff38ac088918093b68eeac3173ad31c,dff38ac0aa72639615964e432c251cf9,,2016-01-15,USD,dff38ac0dfa53a25cf026658c1631e92,dff38ac059b7dd5e83ff0167914620e7,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000005,,2016-01-14,Standard Adjustment,,dff38ac06e013871256e292f2584dc34,dff38ac01e7be6b9ccabb275d0c0d355,INV00000062,Invoice,Processed,EUR,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:46.000000
+dff38ac0bf5884961339694eaef29030,FALSE,2023-02-20 23:03:01.187000,dff38ac075b0b9737438a32e304763fe,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,11212.9,,,dff38ac0402fac05f894822c7262d5fe,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:33.000000,dff38ac088918093b68eeac3173ad31c,dff38ac089b849bd637f8280de505751,,2016-01-15,USD,dff38ac0f3f8366b90ae206d1d20ab4e,dff38ac05d3da6c9e3d7df3eec93ed0c,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000004,,2016-01-14,Standard Adjustment,,dff38ac0402fac05f894822c7262d5fe,dff38ac0e5a5c1258835301ece425def,INV00000061,Invoice,Processed,GBP,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:33.000000
+dff38ac015cf0d852d3dfff06e63a038,FALSE,2023-02-20 23:03:01.187000,dff38ac05ae066e479f6ba9b68a23d47,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,3130859,,,dff38ac07e977c34b06543bd339ddbab,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000,dff38ac088918093b68eeac3173ad31c,dff38ac0359bb2514a0791d861546823,,2016-01-15,USD,dff38ac0ea5bbc65bce34ebdb935e174,dff38ac08d0d6254630cc88ff7301563,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000003,,2016-01-14,Standard Adjustment,,dff38ac07e977c34b06543bd339ddbab,dff38ac0bc55aae22a3d521c8c184d44,INV00000060,Invoice,Processed,JPY,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000
+dff38ac015cf0d852d3dfff06e63a039,FALSE,2023-02-20 23:03:01.187000,dff38ac05ae066e479f6ba9b68a23d47,dff38ac0507cfccf7b1585994a315868,Customer Cash on Account,dff38ac0a66b2bdc2afcc5f5a82a22cb,2016-01-15,3130859,,,dff38ac07e977c34b06543bd339ddbab,,,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000,dff38ac088918093b68eeac3173ad31c,dff38ac0359bb2514a0791d861546823,,2016-01-15,USD,dff38ac0ea5bbc65bce34ebdb935e174,dff38ac08d0d6254630cc88ff7301563,dff38ac041eaf8be7ab4ecbf9f15f3f5,CBA-00000003,,2016-01-14,Standard Adjustment,,dff38ac07e977c34b06543bd339ddbab,dff38ac0bc55aae22a3d521c8c184d44,INV00000060,Invoice,Processed,JPY,,Increase,dff38ac0c94b11b94a7a297293d10e08,2016-01-15 23:13:20.000000