Feature/add conversions

CHANGELOG.md

@@ -1,3 +1,19 @@
+# dbt_facebook_ads_source v0.8.0
+
+[PR #36](https://github.com/fivetran/dbt_facebook_ads_source/pull/36) includes the following updates:
+
+## 🚨 Breaking Changes 🚨
+- Incorporates the `basic_ad_actions` pre-built report in order to grab conversion data. `basic_ad_actions` is a child table of the already-required `basic_ad` report, broken down by `action_type`. 
+  - Addition of the new `stg_facebook_ads__basic_ad_actions` (and its `_tmp` counterpart) staging model. Given that this is a schema change for the package, this a breaking change.
+
+## Feature Updates
+- With the inclusion of the `basic_ad_actions` source table, creates a `facebook_ads__basic_ad_actions_passthrough_metrics` variable to pass through additional conversion value metrics to downstream models. By default, the package includes only the conversion value calculated using the default attribution window, but your report may include calculations using the other windows defined [here](https://developers.facebook.com/docs/marketing-api/reference/ads-action-stats/). See [README](https://github.com/fivetran/dbt_facebook_ads_source/tree/main?tab=readme-ov-file#passing-through-additional-metrics) for details on how to use the new var.
+- Adds `optimization_goal` field to `stg_facebook_ads__ad_set_history` model. This is defined as the optimization goal this ad set is using, possible values of which are defined [here](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign/#fields).
+- Adds `conversion_domain` field to `stg_facebook_ads__ad_history` model. This is defined as the domain you've configured the ad to convert to.
+
+## Documentation
+- Documents the ability to transform metrics provided to the `facebook_ads__basic_ad_passthrough_metrics` variable. See [README](https://github.com/fivetran/dbt_facebook_ads_source/tree/main?tab=readme-ov-file#passing-through-additional-metrics) for details.
+
 # dbt_facebook_ads_source v0.7.3
 [PR #35](https://github.com/fivetran/dbt_facebook_ads_source/pull/35) includes the following updates:
 

README.md

@@ -29,9 +29,9 @@
 To use this dbt package, you must have the following:
 - At least one Fivetran Facebook Ads connector syncing data into your destination. 
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
-- You will need to configure your Facebook Ads connector to pull the `basic_ad` pre-built report. This pre-built report should be enabled in your connector by default. However, to confirm this pre-built report is actively syncing you may perform the following steps:
-    1. Navigate to the connector schema tab.
-    2. Search for `basic_ad` and confirm it is selected.
+- You will need to configure your Facebook Ads connector to pull the `basic_ad` pre-built report and its child `basic_ad_actions` pre-built report. These pre-built reports should be enabled in your connector by default. However, to confirm these reports are actively syncing you may perform the following steps:
+    1. Navigate to the connector schema tab in Fivetran.
+    2. Search for `basic_ad` and `basic_ad_actions` and confirm they are both selected/enabled.
     3. If not selected, do so and sync. If already selected you are ready to run the models!
 
 ### Databricks Dispatch Configuration
@@ -42,13 +42,13 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 ```
 
-## Step 2: Install the package (skip if also using the `facebook_ads` transformation package)
+## Step 2: Install the package (skip if also using the `facebook_ads` transformation or `ad_reporting` combo package)
 Include the following facebook_ads_source package version in your `packages.yml` file.
 > TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
-```yaml
+```yml
 packages:
   - package: fivetran/facebook_ads_source
-    version: [">=0.7.0", "<0.8.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 ## Step 3: Define database and schema variables
 By default, this package runs using your destination and the `facebook_ads` schema. If this is not where your Facebook Ads data is (for example, if your facebook_ads schema is named `facebook_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -73,17 +73,25 @@ Please be aware that the native `source.yml` connection set up in the package wi
 To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
 
 ### Passing Through Additional Metrics
-By default, this package will select `clicks`, `impressions`, and `cost` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) if desired, but not required. Use the below format for declaring the respective pass-through variables:
-
->**Note** Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (taps, impressions, and spend) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+By default, this package will select `clicks`, `impressions`, `cost`, and conversion `value` (using the default attribution window) from the source reporting tables (`BASIC_AD` and `BASIC_AD_ACTIONS`) to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) and transformed (`transform_sql`) if desired, but not required. Only the `name` of each metric field is required. Use the below format for declaring the respective pass-through variables:
 
 ```yml
 vars:
-    facebook_ads__basic_ad_passthrough_metrics: 
+    facebook_ads__basic_ad_passthrough_metrics: # add metrics found in BASIC_AD
       - name: "new_custom_field"
-        alias: "custom_field"
+        alias: "custom_field_alias"
+        transform_sql: "coalesce(custom_field_alias, 0)" # reference the `alias` here if you are using one
       - name: "another_one"
+        transform_sql: "coalesce(another_one, 0)" # reference the `name` here if you're not using an alias
+      - name: "cpc" # only name is required
+    facebook_ads__basic_ad_actions_passthrough_metrics: # add conversion metrics found in BASIC_AD_ACTIONS
+      - name: "_7_d_click"
+        alias: "conversion_value_7d_click"
+      - name: "inline"
 ```
+
+>**Note** Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (taps, impressions, spend, and default-attribution window conversion values) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+
 ### Change the build schema
 By default, this package builds the Facebook Ads staging models within a schema titled (`<target_schema>` + `_facebook_ads_source`) in your destination. If this is not where you would like your Facebook Ads staging data to be written to, add the following configuration to your root `dbt_project.yml` file:
 
@@ -94,7 +102,8 @@ models:
 ```
 
 ### Change the source table references
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable. This is not available when running the package on multiple unioned connectors.
+
 > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_facebook_ads_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
 ```yml
@@ -135,5 +144,4 @@ We highly encourage and welcome contributions to this package. Check out [this d
 
 # 🏪 Are there any resources available?
 - If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_facebook_ads_source/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
-- Have questions or want to be part of the community discourse? Create a post in the [Fivetran community](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) and our team along with the community can join in on the discussion!
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'facebook_ads_source'
-version: '0.7.3'
+version: '0.8.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
@@ -14,6 +14,9 @@ vars:
     ad_history: "{{ source('facebook_ads','ad_history') }}"
     ad_set_history: "{{ source('facebook_ads','ad_set_history') }}"
     basic_ad: "{{ source('facebook_ads','basic_ad') }}"
+    basic_ad_actions: "{{ source('facebook_ads','basic_ad_actions') }}"
     campaign_history: "{{ source('facebook_ads','campaign_history') }}"
     creative_history: "{{ source('facebook_ads','creative_history') }}"
+
   facebook_ads__basic_ad_passthrough_metrics: []
+  facebook_ads__basic_ad_actions_passthrough_metrics: []

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'facebook_ads_source_integration_tests_2'
-version: '0.7.3'
+version: '0.8.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -8,13 +8,20 @@ vars:
     - name: "reach"
     - name: "frequency"
       alias: "fake_freq_name"
-
+  facebook_ads__basic_ad_actions_passthrough_metrics: 
+    - name: 'inline'
+      alias: 'conversion_value_inline'
+    - name: '_1_d_view'
+    - name: '_7_d_click'
+    - name: '_28_d_click'
+
   facebook_ads_schema: facebook_ads_source_integration_tests_2
   facebook_ads_source:
     facebook_ads_account_history_identifier: "facebook_ads_account_history_data"
     facebook_ads_ad_history_identifier: "facebook_ads_ad_history_data"
     facebook_ads_ad_set_history_identifier: "facebook_ads_ad_set_history_data"
     facebook_ads_basic_ad_identifier: "facebook_ads_basic_ad_data"
+    facebook_ads_basic_ad_actions_identifier: "facebook_ads_basic_ad_actions_data"
     facebook_ads_campaign_history_identifier: "facebook_ads_campaign_history_data"
     facebook_ads_creative_history_identifier: "facebook_ads_creative_history_data"
 
@@ -31,6 +38,12 @@ seeds:
       template_page_link: "{{ 'string' if target.type in ['bigquery','spark','databricks'] else 'varchar' }}"
       _fivetran_synced: "timestamp"
       updated_time: "timestamp"
+    facebook_ads_basic_ad_actions_data:
+      +column_types:
+        value: "float"
+        inline: "float"
+        _1_d_view: "float"
+        _7_d_click: "float"
 
 dispatch:
   - macro_namespace: dbt_utils

integration_tests/seeds/facebook_ads_basic_ad_actions_data.csv

@@ -0,0 +1,11 @@
+_fivetran_id,ad_id,date,index,_1_d_view,_7_d_click,_fivetran_synced,action_type,inline,value
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-24,0,2.0,,2023-05-18 18:41:01.193000,offsite_conversion.fb_pixel_lead,,2.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-24,1,2.0,,2023-05-18 18:41:01.193000,lead,,2.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-03,0,,2.0,2023-05-18 18:40:49.736000,offsite_conversion.fb_pixel_lead,,2.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-23,0,,2.0,2023-05-18 18:41:01.192000,offsite_conversion.fb_pixel_lead,,2.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-11-21,1,,2.0,2023-05-18 18:40:42.369000,offsite_conversion.fb_pixel_lead,,2.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-15,0,,3.0,2023-05-18 18:40:49.761000,offsite_conversion.fb_pixel_lead,,3.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-12-28,0,,3.0,2023-06-28 10:58:39.102000,offsite_conversion.fb_pixel_lead,,3.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317634610871,2022-11-19,0,,4.0,2023-05-18 18:40:42.358000,offsite_conversion.fb_pixel_lead,,4.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317039926871,2022-11-20,0,,1.0,2023-05-18 18:40:42.361000,offsite_conversion.fb_pixel_lead,,1.0
+2jmj7l5rSw0yVb/vlWAYkK/YBwk=,6317039926871,2022-11-20,1,,1.0,2023-05-18 18:40:42.361000,lead,,1.0

macros/get_ad_history_columns.sql

@@ -7,7 +7,8 @@
     {"name": "account_id", "datatype": dbt.type_int()},
     {"name": "ad_set_id", "datatype": dbt.type_int()},
     {"name": "campaign_id", "datatype": dbt.type_int()},
-    {"name": "creative_id", "datatype": dbt.type_int()}
+    {"name": "creative_id", "datatype": dbt.type_int()},
+    {"name": "conversion_domain", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

macros/get_ad_set_history_columns.sql

@@ -11,7 +11,8 @@
     {"name": "bid_strategy", "datatype": dbt.type_string()},
     {"name": "daily_budget", "datatype": dbt.type_int()},
     {"name": "budget_remaining", "datatype": dbt.type_int()},
-    {"name": "status", "datatype": dbt.type_string()}
+    {"name": "status", "datatype": dbt.type_string()},
+    {"name": "optimization_goal", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

macros/get_basic_ad_actions_columns.sql

@@ -0,0 +1,17 @@
+{% macro get_basic_ad_actions_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_id", "datatype": dbt.type_string()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "action_type", "datatype": dbt.type_string()},
+    {"name": "ad_id", "datatype": dbt.type_string()},
+    {"name": "date", "datatype": "date"},
+    {"name": "index", "datatype": dbt.type_int()},
+    {"name": "value", "datatype": dbt.type_float()}
+] %}
+
+{{ fivetran_utils.add_pass_through_columns(columns, var('facebook_ads__basic_ad_actions_passthrough_metrics')) }}
+
+{{ return(columns) }}
+
+{% endmacro %}

models/src_facebook_ads.yml

@@ -57,6 +57,8 @@ sources:
             description: "{{ doc('_fivetran_synced') }}"
           - name: updated_time
             description: "{{ doc('updated_time') }}"
+          - name: conversion_domain
+            description: The domain you've configured the ad to convert to.
 
       - name: ad_set_history
         description: Each record in this table reflects a version of a Facebook ad set.
@@ -86,6 +88,8 @@ sources:
             description: Remaining budget of ad set.
           - name: status
             description: Status values are - 'ACTIVE', 'PAUSED', 'DELETED', 'ARCHIVED'.
+          - name: optimization_goal
+            description: The optimization goal this ad set is using. Possible values defined [here](https://developers.facebook.com/docs/marketing-api/reference/ad-campaign/#fields).
 
       - name: basic_ad
         description: Each record represents the daily performance of an ad in Facebook.
@@ -114,6 +118,35 @@ sources:
           - name: _fivetran_synced
             description: "{{ doc('_fivetran_synced') }}"
 
+      - name: basic_ad_actions
+        description: Each record represents the daily conversion performance of an ad in Facebook. This is the prebuilt `basic_ad` report broken down by `action_type`.
+        identifier: "{{ var('facebook_ads_basic_ad_actions_identifier', 'basic_ad_actions') }}"
+        columns:
+          - name: _1_d_view
+            description: Conversion metric value using an attribution window of "1 day after viewing the ad". Not included in downstream models by default, but can be added using the `facebook_ads__basic_ad_actions_passthrough_metrics` var.
+          - name: _7_d_click
+            description: Conversion metric value using an attribution window of "7 days after clicking the ad". Not included in downstream models by default, but can be added using the `facebook_ads__basic_ad_actions_passthrough_metrics` var.
+          - name: _28_d_click
+            description: Conversion metric value using an attribution window of "28 days after clicking the ad". Deprecated by Facebook due to digital privacy initiatives. Not included in downstream models by default, but can be added using the `facebook_ads__basic_ad_actions_passthrough_metrics` var.
+          - name: _fivetran_id
+            description: Defunct field.
+          - name: _fivetran_synced
+            description: "{{ doc('_fivetran_synced') }}"
+          - name: action_type
+            description: >
+              The kind of actions taken on your ad, Page, app or event after your ad was served to someone, even if they didn't click on it. Action types include Page likes, app installs, conversions, event responses and more.
+              Actions prepended by app_custom_event come from mobile app events and actions prepended by offsite_conversion come from the Facebook Pixel.
+          - name: ad_id
+            description: The ID of the ad the report relates to.
+          - name: date
+            description: The date of the reported performance.
+          - name: index
+            description: Index reflecting the `action_type` tracked for this ad on this day. Column of not much consequence. 
+          - name: inline
+            description: Conversion metric value using the attribution window that occurs on the ad itself. Not included in downstream models by default, but can be added using the `facebook_ads__basic_ad_actions_passthrough_metrics` var.
+          - name: value
+            description: Conversion metric value using the default attribution window.
+
       - name: campaign_history
         description: Each record in this table reflects a version of a Facebook campaign.
         identifier: "{{ var('facebook_ads_campaign_history_identifier', 'campaign_history') }}"