Add conversions to Linkedin Ads Transform package (#36)

* Update models, update documentation, create testing branch

* new passthrough logic

* updated documentation, seed files, macros, regen docs

* syntax and schema

* readme

* DECISIONLOG & README

* PR fixes

* Regn docs

* changelog

* changelog

* update docs with code update

* update source pkg ref

* joe suggestion

---------

Co-authored-by: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com>

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,22 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present)
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
-<!--- Provide the steps you took to validate your changes below. -->
-
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
-### If you had to summarize this PR in an emoji, which would it be?
-<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+Please share any and all of your validation steps:
+<!--- Provide the steps you took to validate your changes below. -->

.github/workflows/auto-release.yml

@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit

.gitignore

@@ -1,5 +1,7 @@
 target/
+env/
 dbt_modules/
 logs/
 .DS_Store
-dbt_packages/
+dbt_packages/
+package-lock.yml

CHANGELOG.md

@@ -1,3 +1,30 @@
+# dbt_linkedin v0.9.0
+[PR #36](https://github.com/fivetran/dbt_linkedin/pull/36) includes the following updates:
+
+## Feature Updates: Conversion Support!
+We have added more robust support for conversions in our data models by doing the following: 
+- Adds a `conversion_value_by_local_currency` field to each `_report` end model, representing the value of conversions that occurred on each day for each campaign/campaign_group/creative/url/account.
+- Created a `linkedin_ads__conversion_fields` variable to pass through and transform additional conversion value metrics into their aggregate sums.
+  - Set current variable defaults in the `dbt_project.yml` to bring in the most used conversion fields `external_website_conversions` and `one_click_leads` by default.
+  - Instructions on how to set your own fields [are available in the README](https://github.com/fivetran/dbt_linkedin/blob/main/README.md#adding-in-conversion-fields-variable).
+- Adds a `total_conversions` metric in our end models to track all conversions being brought in by the `linkedin_ads_conversion_fields` variable.
+> The above new field additions are 🚨 **breaking changes** 🚨.
+
+## Documentation Update 
+- Documents how to set your own passthrough fields with the variable `linkedin_ads__conversion_fields` [in the README](https://github.com/fivetran/dbt_linkedin_source/blob/main/README.md#adding-in-conversion-fields-variable).
+- [Created a DECISIONLOG](https://github.com/fivetran/dbt_linkedin/blob/main/DECISIONLOG.md#best-practices-with-configuring-linkedin-ads-conversion-fields-variable) to describe best practices in configuring the `linkedin_ads__conversion_fields` variable. 
+
+## Under the Hood
+- Added a new [version](https://github.com/fivetran/dbt_linkedin_ads/blob/main/macros/linkedin_ads_persist_pass_through_columns.sql) of the `persist_pass_through_columns()` [macro](https://github.com/fivetran/dbt_fivetran_utils/blob/v0.4.10/macros/persist_pass_through_columns.sql) in which we can include `coalesces` and properly check between conversion field values and the existing passthrough column.
+- Updated the PR templates to align with our most up-to-date standards.
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+- Addition of integrity and consistency validation tests within `integration_tests` for the Linkedin transformation models.
+- Updated `linkedin_ad_analytics_by_creative_data` seed file with relevant conversion fields for more robust testing. 
+
+## Contributors
+- [Seer Interactive](https://www.seerinteractive.com/?utm_campaign=Fivetran%20%7C%20Models&utm_source=Fivetran&utm_medium=Fivetran%20Documentation)
+
+
 # dbt_linkedin v0.8.0
 [PR #32](https://github.com/fivetran/dbt_linkedin/pull/32) includes the following updates:
 

DECISIONLOG.md

@@ -0,0 +1,24 @@
+## Best Practices with Configuring LinkedIn Ads Conversion Fields Variable
+The `linkedin_ads__conversion_fields` variable is designed for end users to properly measure the conversions at the proper level of granularity. By default, we use `external_website_conversions` and `one_click_leads` as they are arguably the most used conversion measures, and fulfill entirely separate objectives as conversions (Website Conversion and Lead Generation respectively). 
+
+However, if you decide to configure your own conversion field variable fields, we highly recommend that you bring in conversions at the proper level of segmentation, so there aren't conversions that belong to multiple fields you bring in.
+
+### Bad Practice Example
+
+```yml
+# dbt_project.yml
+vars:
+    linkedin_ads__conversion_fields: ['external_website_conversions', 'external_website_pre_click_conversions', 'external_website_post_click_conversions']
+```
+
+`external_website_conversions` is comprised of both `external_website_pre_click_conversions` and `external_website_post_click_conversions`, so `total_conversions` in the end models would be double counting conversions. 
+
+### Good Practice Example
+
+```yml
+# dbt_project.yml
+vars:
+    linkedin_ads__conversion_fields: ['external_website_pre_click_conversions', 'external_website_post_click_conversions']
+```
+
+`external_website_pre_click_conversions` and `external_website_post_click_conversions` are two different type of external website conversions, so there should be no overlap. 

README.md

@@ -48,14 +48,14 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 ```
 
-## Step 2: Install the package
+## Step 2: Install the package (skip if also using the `ad_reporting` combination package)
 Include the following Linkedin Ads package version in your `packages.yml` file:
 > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages
 ```yml
 # packages.yml
 packages:
   - package: fivetran/linkedin
-    version: [">=0.8.0", "<0.9.0"]
+    version: [">=0.9.0", "<0.10.0"]
 ```
 Do **NOT** include the `linkedin_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. 
 
@@ -91,25 +91,46 @@ vars:
     linkedin_ads__use_local_currency: True # false by default -- uses USD
 ```
 
-### Passing Through Additional Metrics
-By default, this package will select `clicks`, `impressions`, and `cost` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) if desired, but not required. Use the below format for declaring the respective pass-through variables:
+**Note**: Unlike cost, conversion values are only available in the local currency. The package will only use the `conversion_value_in_local_currency` field for conversion values, while it may draw from the `cost_in_local_currency` and `cost_in_usd` source fields for cost.
 
->**Note** Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (taps, impressions, and spend) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+### Passing Through Additional Metrics
+By default, this package will select `clicks`, `impressions`, `cost`, `conversion_value_in_local_currency`, and `total_conversions` (as well as fields set via `linkedin_ads__conversion_fields` in the next section) from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) if desired, but not required. Use the below format for declaring the respective pass-through variables:
 
 ```yml
 # dbt_project.yml
 vars:
     linkedin_ads__campaign_passthrough_metrics: # pulls from ad_analytics_by_campaign
         - name: "new_custom_field"
-          alias: "custom_field"
+          alias: "custom_field_alias"
+          transform_sql: "coalesce(custom_field_alias, 0)" # reference the `alias` here if you are using one
         - name: "unique_int_field"
           alias: "field_id"
+        - name: "another_one"
+          transform_sql: "coalesce(another_one, 0)" # reference the `name` here if you're not using an alias
         - name: "that_field"
     linkedin_ads__creative_passthrough_metrics: # pulls from ad_analytics_by_creative
         - name: "new_custom_field"
           alias: "custom_field"
         - name: "unique_int_field"
 ```
+
+>**Note** Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (clicks, impressions, and spend) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package. (**Important**: You do not need to add conversions in this way. See the following section for an alternative implementation.)
+
+### Adding in Conversion Fields Variable
+Separate from the above passthrough metrics, the package will also include conversion metrics based on the `linkedin_ads__conversion_fields` variable, in addition to the `conversion_value_in_local_currency` field.
+
+By default, the data models consider `external_website_conversions` and `one_click_leads` to be conversions. These should cover most use cases, but if you wanted to adjust this to your business case, you would apply the following configuration with the **original** source names of the conversion fields (not aliases you provided in the section above):
+
+```yml
+# dbt_project.yml
+vars:
+    linkedin_ads__conversion_fields: ['external_website_pre_click_conversions', 'one_click_leads', 'external_website_post_click_conversions', 'landing_page_clicks']
+```
+
+Make sure to follow best practices in configuring fields in the conversion field variables! [See the DECISIONLOG for more details](https://github.com/fivetran/dbt_linkedin/blob/main/DECISIONLOG.md#best-practices-with-configuring-linkedin-ads-conversion-fields-variable). 
+
+> We introduced support for conversion fields in our `report` data models in the [v0.9.0 release](https://github.com/fivetran/dbt_linkedin/releases/tag/v0.9.0) of the package, but customers might have been bringing in these conversion fields earlier using the passthrough fields variables. The data models will avoid "duplicate column" errors automatically if this is the case. 
+
 ### Changing the Build Schema
 By default this package will build the LinkedIn Ad Analytics staging models within a schema titled (<target_schema> + `_linkedin_ads_source`) and the LinkedIn Ad Analytics final models within a schema titled (<target_schema> + `_linkedin_ads`) in your target database. If this is not where you would like your modeled LinkedIn data to be written to, add the following configuration to your `dbt_project.yml` file:
 
@@ -123,7 +144,8 @@ models:
 ```
 
 ### Change the source table references
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable. This is not available when running the package on multiple unioned connectors.
+
 > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_linkedin/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
 ```yml
@@ -145,7 +167,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/linkedin_source
-      version: [">=0.8.0", "<0.9.0"]
+      version: [">=0.9.0", "<0.10.0"]
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]
     - package: dbt-labs/dbt_utils

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'linkedin'
-version: '0.8.0'
+version: '0.9.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:
@@ -12,6 +12,8 @@ vars:
     ad_analytics_by_campaign: "{{ ref('stg_linkedin_ads__ad_analytics_by_campaign') }}"
   linkedin_ads__creative_passthrough_metrics: []
   linkedin_ads__campaign_passthrough_metrics: []
+  linkedin_ads__conversion_fields: ['external_website_conversions', 'one_click_leads']
+
 models:
   linkedin:
     +materialized: table