Feature/click uri deprecation (#69)

* feature/click-uri-deprecation

* feature/click-uri-deprecation

* feature/click-uri-deprecation

* full code changes and docs regen

* docs regen

* docs fix

* changelog update

* Update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

* review updates

* more explicit macro adjustment

* complex macro removal

* jinja formatting fix

* window function move

* docs regen

* changelog review update

---------

Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com>
Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

CHANGELOG.md

@@ -1,3 +1,29 @@
+# dbt_linkedin_source v0.10.0
+[PR #69](https://github.com/fivetran/dbt_linkedin_source/pull/69) includes the following updates:
+
+## Breaking Changes
+- The `click_uri_type` field has been added to the `stg_linkedin_ads__creative_history` model. This field allows users to differentiate which click uri type (`text_ad` or `spotlight`) is being used to populate the results of the `click_uri` field. 
+  - Please be aware this field only supports `text_ad` or `spotlight` click uri types. If you are interested in this package supporting more click uri ad types, please let us know in this [Feature Request](https://github.com/fivetran/dbt_linkedin_source/issues/70).
+
+## Bug Fixes
+- The `click_uri` field has been adjusted to populate the results following a coalesce on the `text_ad_landing_page`, `spotlight_landing_page`, or `click_uri` fields. 
+  - This change is in response to a [LinkedIn Ads API](https://learn.microsoft.com/en-us/linkedin/marketing/community-management/contentapi-migration-guide?view=li-lms-2024-05#adcreativesv2-api-creatives-api) and [Fivetran LinkedIn Ads connector update](https://fivetran.com/docs/connectors/applications/linkedin-ads/changelog#january2024) which moved `click_uri` data to either the `text_ad_landing_page` or `spotlight_landing_page` fields depending on the creative type.
+- Updated the `is_latest_version` window function in the following models to exclude the `source_relation` field from the partition statement when `linkedin_ads_union_schemas` or `linkedin_ads_union_databases` variables are empty in the following models:
+    - `stg_linkedin_ads__account_history`
+    - `stg_linkedin_ads__campaign_group_history`
+    - `stg_linkedin_ads__campaign_history`
+    - `stg_linkedin_ads__creative_history`
+- In addition to the above, the `is_latest_version` window function within the `stg_linkedin_ads__creative_history` model has been moved to the final cte to avoid possible constant expression errors within Redshift destinations.
+
+## Under the Hood
+- Updates to the `linkedin_creative_history_data` seed file to include the following new fields to ensure accurate data validation tests:
+  - `text_ad_landing_page`
+  - `spotlight_landing_page`
+
+## Documentation Updates
+- Added `click_uri_type` field documentation.
+- The `click_uri` field documentation has been updated to reflect the updated state of the field.
+
 # dbt_linkedin_source v0.9.0
 [PR #67](https://github.com/fivetran/dbt_linkedin_source/pull/67) includes the following updates:
 

README.md

@@ -47,7 +47,7 @@ If you  are **not** using the [Linkedin transformation package](https://github.c
 # packages.yml
 packages:
   - package: fivetran/linkedin_source
-    version: [">=0.9.0", "<0.10.0"]
+    version: [">=0.10.0", "<0.11.0"]
 ```
 
 ### Step 3: Define database and schema variables

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'linkedin_source'
-version: '0.9.0'
+version: '0.10.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'linkedin_source_integration_tests'
-version: '0.9.0'
+version: '0.10.0'
 profile: 'integration_tests'
 config-version: 2
 

integration_tests/seeds/linkedin_creative_history_data.csv

@@ -1,101 +1,101 @@
-id,campaign_id,intended_status,click_uri,last_modified_at,created_time
-125339904,174942134,,,,2021-03-08 23:42:20
-186286604,202434464,,,,2022-09-16 02:46:54
-54106906,131454616,,,,2018-05-18 01:04:02
-54106976,131454826,,,,2018-05-18 01:09:08
-54299976,131580446,,,,2018-05-24 00:38:47
-54300176,131580546,,,,2018-05-24 00:47:40
-82332766,148633856,,,,2020-03-20 18:55:40
-82460426,148711996,,,,2020-03-24 17:56:49
-102541764,162464924,,,,2020-05-08 09:34:15
-102694624,162557074,,,,2020-05-12 14:08:40
-107943663,165867684,,,,2020-07-22 19:28:31
-110663234,167276694,,,,2020-09-10 18:08:57
-44900356,125600246,,,,2017-06-08 21:46:06
-44900356,125600246,,,,2017-06-08 21:46:06
-44900356,125600246,,,,2017-06-08 21:46:06
-44900356,125600246,,,,2017-06-08 21:46:06
-44900336,125600246,,,,2017-06-08 21:46:06
-44900336,125600246,,,,2017-06-08 21:46:06
-44900336,125600246,,,,2017-06-08 21:46:06
-44900336,125600246,,,,2017-06-08 21:46:06
-44900366,125600246,,,,2017-06-08 21:46:06
-44900366,125600246,,,,2017-06-08 21:46:06
-44900366,125600246,,,,2017-06-08 21:46:06
-44900366,125600246,,,,2017-06-08 21:46:06
-44900346,125600246,,,,2017-06-08 21:46:06
-44900346,125600246,,,,2017-06-08 21:46:06
-44900346,125600246,,,,2017-06-08 21:46:06
-44900346,125600246,,,,2017-06-08 21:46:06
-55220334,133160324,,,,2017-08-04 22:45:09
-55220334,133160324,,,,2017-08-04 22:45:09
-44901856,125599426,,,,2017-06-08 22:29:09
-55220334,133160324,,,,2017-08-04 22:45:09
-44901856,125599426,,,,2017-06-08 22:29:09
-44901856,125599426,,,,2017-06-08 22:29:09
-44901866,125599426,,,,2017-06-08 22:29:09
-44901866,125599426,,,,2017-06-08 22:29:09
-44901866,125599426,,,,2017-06-08 22:29:09
-55220324,133160324,,,,2017-08-04 22:45:09
-55220324,133160324,,,,2017-08-04 22:45:09
-55220324,133160324,,,,2017-08-04 22:45:09
-44901876,125599426,,,,2017-06-08 22:29:09
-55220314,133160324,,,,2017-08-04 22:45:09
-44901876,125599426,,,,2017-06-08 22:29:09
-55220314,133160324,,,,2017-08-04 22:45:09
-55220314,133160324,,,,2017-08-04 22:45:09
-44901876,125599426,,,,2017-06-08 22:29:09
-44901876,125599426,,,,2017-06-08 22:29:09
-55220314,133160324,,,,2017-08-04 22:45:09
-55220304,133160324,,,,2017-08-04 22:45:09
-44901886,125599426,,,,2017-06-08 22:29:09
-44901886,125599426,,,,2017-06-08 22:29:09
-44901886,125599426,,,,2017-06-08 22:29:09
-55220304,133160324,,,,2017-08-04 22:45:09
-55220304,133160324,,,,2017-08-04 22:45:09
-55220304,133160324,,,,2017-08-04 22:45:09
-44901886,125599426,,,,2017-06-08 22:29:09
-48046336,127624356,,,,2017-10-10 16:49:38
-48046326,127624356,,,,2017-10-10 16:49:38
-48046346,127624356,,,,2017-10-10 16:49:38
-50036216,128806186,,,,2017-12-22 17:38:12
-50035816,128806016,,,,2017-12-22 17:27:45
-50036226,128806186,,,,2017-12-22 17:38:12
-50035806,128806016,,,,2017-12-22 17:27:45
-56166306,132822186,,,,2018-07-25 15:04:05
-56166306,132822186,,,,2018-07-25 15:04:05
-56166306,132822186,,,,2018-07-25 15:04:05
-56237466,132870856,,,,2018-07-28 18:48:26
-56237376,132866546,,,,2018-07-28 18:38:10
-57434376,133672706,,,,2018-09-06 16:59:15
-57527536,133728466,,,,2018-09-10 17:52:28
-57542836,133736786,,,,2018-09-10 23:16:13
-57675346,133822116,,,,2018-09-13 17:59:19
-58503736,134383046,,,,2018-10-08 17:49:29
-58503856,134383176,,,,2018-10-08 17:54:17
-59428846,134988516,,,,2018-10-31 17:30:40
-60400066,135583676,,,,2018-11-21 17:08:53
-61088626,136022286,,,,2018-12-10 18:01:42
-61884186,136508426,,,,2019-01-08 17:27:32
-61885366,136509566,,,,2019-01-08 17:57:44
-62024726,136598756,,,,2019-01-11 21:00:57
-62257986,136746346,,,,2019-01-17 19:19:05
-63506906,137527956,,,,2019-02-20 17:23:30
-63556186,137559956,,,,2019-02-21 17:46:17
-64024086,137824786,,,,2019-03-04 19:12:56
-66558706,139304236,,,,2019-04-29 17:20:56
-66559306,139304466,,,,2019-04-29 17:32:14
-66980846,139548496,,,,2019-05-08 17:34:16
-67039296,139581266,,,,2019-05-09 15:19:05
-67131396,139619466,,,,2019-05-10 16:05:30
-67495816,139812856,,,,2019-05-17 14:50:20
-68192386,140203726,,,,2019-06-03 19:34:54
-68197356,140206956,,,,2019-06-03 20:59:46
-68755776,140506806,,,,2019-06-14 17:17:18
-69108056,140715576,,,,2019-06-21 18:33:19
-69775776,141092076,,,,2019-07-08 15:16:08
-69775486,141091746,,,,2019-07-08 15:13:19
-69776656,141092526,,,,2019-07-08 15:24:18
-69777076,141092656,,,,2019-07-08 15:29:15
-70247496,141360566,,,,2019-07-18 15:40:03
-70668196,141360566,,,,2019-07-29 18:33:05
+id,campaign_id,intended_status,click_uri,last_modified_at,created_time,text_ad_landing_page,spotlight_landing_page
+125339904,174942134,,,,2021-03-08 23:42:20,,"http://www.linkedin.com"
+186286604,202434464,,,,2022-09-16 02:46:54,,"http://www.linkedin.com"
+54106906,131454616,,,,2018-05-18 01:04:02,,"http://www.linkedin.com"
+54106976,131454826,,,,2018-05-18 01:09:08,,"http://www.linkedin.com"
+54299976,131580446,,,,2018-05-24 00:38:47,,"http://www.linkedin.com"
+54300176,131580546,,,,2018-05-24 00:47:40,,"http://www.linkedin.com"
+82332766,148633856,,,,2020-03-20 18:55:40,,"http://www.linkedin.com"
+82460426,148711996,,,,2020-03-24 17:56:49,,"http://www.linkedin.com"
+102541764,162464924,,,,2020-05-08 09:34:15,,"http://www.linkedin.com"
+102694624,162557074,,,,2020-05-12 14:08:40,,"http://www.linkedin.com"
+107943663,165867684,,,,2020-07-22 19:28:31,,"http://www.linkedin.com"
+110663234,167276694,,,,2020-09-10 18:08:57,,"http://www.linkedin.com"
+44900356,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900356,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900356,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900356,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900336,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900336,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900336,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900336,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900366,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900366,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900366,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900366,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900346,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900346,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900346,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+44900346,125600246,,,,2017-06-08 21:46:06,,"http://www.linkedin.com"
+55220334,133160324,,,,2017-08-04 22:45:09,,"http://www.linkedin.com"
+55220334,133160324,,,,2017-08-04 22:45:09,,"http://www.linkedin.com"
+44901856,125599426,,,,2017-06-08 22:29:09,,"http://www.linkedin.com"
+55220334,133160324,,,,2017-08-04 22:45:09,,"http://www.linkedin.com"
+44901856,125599426,,,,2017-06-08 22:29:09,,"http://www.linkedin.com"
+44901856,125599426,,,,2017-06-08 22:29:09,,"http://www.linkedin.com"
+44901866,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+44901866,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+44901866,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+55220324,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220324,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220324,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+44901876,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+55220314,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+44901876,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+55220314,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220314,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+44901876,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+44901876,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+55220314,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220304,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+44901886,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+44901886,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+44901886,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+55220304,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220304,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+55220304,133160324,,,,2017-08-04 22:45:09,"http://www.google.com",
+44901886,125599426,,,,2017-06-08 22:29:09,"http://www.google.com",
+48046336,127624356,,,,2017-10-10 16:49:38,"http://www.google.com",
+48046326,127624356,,,,2017-10-10 16:49:38,"http://www.google.com",
+48046346,127624356,,,,2017-10-10 16:49:38,"http://www.google.com",
+50036216,128806186,,,,2017-12-22 17:38:12,"http://www.google.com",
+50035816,128806016,,,,2017-12-22 17:27:45,"http://www.google.com",
+50036226,128806186,,,,2017-12-22 17:38:12,"http://www.google.com",
+50035806,128806016,,,,2017-12-22 17:27:45,"http://www.google.com",
+56166306,132822186,,,,2018-07-25 15:04:05,"http://www.google.com",
+56166306,132822186,,,,2018-07-25 15:04:05,"http://www.google.com",
+56166306,132822186,,,,2018-07-25 15:04:05,"http://www.google.com",
+56237466,132870856,,,,2018-07-28 18:48:26,"http://www.google.com",
+56237376,132866546,,,,2018-07-28 18:38:10,"http://www.google.com",
+57434376,133672706,,,,2018-09-06 16:59:15,"http://www.google.com",
+57527536,133728466,,,,2018-09-10 17:52:28,"http://www.google.com",
+57542836,133736786,,,,2018-09-10 23:16:13,"http://www.google.com",
+57675346,133822116,,"http://www.github.com",,2018-09-13 17:59:19,,
+58503736,134383046,,"http://www.github.com",,2018-10-08 17:49:29,,
+58503856,134383176,,"http://www.github.com",,2018-10-08 17:54:17,,
+59428846,134988516,,"http://www.github.com",,2018-10-31 17:30:40,,
+60400066,135583676,,"http://www.github.com",,2018-11-21 17:08:53,,
+61088626,136022286,,"http://www.github.com",,2018-12-10 18:01:42,,
+61884186,136508426,,"http://www.github.com",,2019-01-08 17:27:32,,
+61885366,136509566,,"http://www.github.com",,2019-01-08 17:57:44,,
+62024726,136598756,,"http://www.github.com",,2019-01-11 21:00:57,,
+62257986,136746346,,"http://www.github.com",,2019-01-17 19:19:05,,
+63506906,137527956,,"http://www.github.com",,2019-02-20 17:23:30,,
+63556186,137559956,,"http://www.github.com",,2019-02-21 17:46:17,,
+64024086,137824786,,"http://www.github.com",,2019-03-04 19:12:56,,
+66558706,139304236,,"http://www.github.com",,2019-04-29 17:20:56,,
+66559306,139304466,,"http://www.github.com",,2019-04-29 17:32:14,,
+66980846,139548496,,"http://www.github.com",,2019-05-08 17:34:16,,
+67039296,139581266,,"http://www.github.com",,2019-05-09 15:19:05,,
+67131396,139619466,,"http://www.github.com",,2019-05-10 16:05:30,,
+67495816,139812856,,"http://www.github.com",,2019-05-17 14:50:20,,
+68192386,140203726,,"http://www.github.com",,2019-06-03 19:34:54,,
+68197356,140206956,,"http://www.github.com",,2019-06-03 20:59:46,,
+68755776,140506806,,"http://www.github.com",,2019-06-14 17:17:18,,
+69108056,140715576,,"http://www.github.com",,2019-06-21 18:33:19,,
+69775776,141092076,,"http://www.github.com",,2019-07-08 15:16:08,,
+69775486,141091746,,"http://www.github.com",,2019-07-08 15:13:19,,
+69776656,141092526,,"http://www.github.com",,2019-07-08 15:24:18,,
+69777076,141092656,,"http://www.github.com",,2019-07-08 15:29:15,,
+70247496,141360566,,"http://www.github.com",,2019-07-18 15:40:03,,
+70668196,141360566,,"http://www.github.com",,2019-07-29 18:33:05,,

macros/get_creative_history_columns.sql

@@ -10,6 +10,8 @@
     {"name": "last_modified_at", "datatype": dbt.type_timestamp()},
     {"name": "intended_status", "datatype": dbt.type_string()},
     {"name": "status", "datatype": dbt.type_string()},
+    {"name": "text_ad_landing_page", "datatype": dbt.type_string()},
+    {"name": "spotlight_landing_page", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

models/src_linkedin.yml

@@ -69,9 +69,13 @@ sources:
             description: The ID of the campaign the creative belongs to.
 
           - name: click_uri
-            description: |
-                Required when creative type is TEXT_AD. Maximum character limit of 500 after URL encoding all special characters. 
-                Limit does not include the "https://" prefix.
+            description: Deprecated url field. Please refer to the text_ad_landing_page and spotlight_landing_page fields for relevant landing page information.  
+
+          - name: text_ad_landing_page
+            description: The URL where the member should be redirected to, on clicking the text ad.
+
+          - name: spotlight_landing_page
+            description: The URL where the member should be redirected to, on clicking the spotlight ad.
 
           - name: intended_status
             description: |

models/stg_linkedin.yml

@@ -111,9 +111,10 @@ models:
           CANCELED - Creative is permanently canceled.
 
       - name: click_uri
-        description: |
-          Required when creative type is TEXT_AD. Maximum character limit of 500 after URL encoding all special characters. 
-          Limit does not include the "https://" prefix.
+        description: The URL where the member should be redirected to, on clicking the text ad or spotlight ad.
+
+      - name: click_uri_type
+        description: Will either be `text_ad`, `spotlight`, or `null` to indicate the origin of the click_uri.
 
       - name: base_url
         description: The base URL of the ad, extracted from the `click_uri`.

models/stg_linkedin_ads__account_history.sql

@@ -34,7 +34,7 @@ with base as (
         type,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by source_relation, id order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/stg_linkedin_ads__campaign_group_history.sql

@@ -35,7 +35,7 @@ with base as (
         cast(run_schedule_end as {{ dbt.type_timestamp() }}) as run_schedule_end_at,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by source_relation, id order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/stg_linkedin_ads__campaign_history.sql

@@ -50,7 +50,7 @@ with base as (
         cast(run_schedule_end as {{ dbt.type_timestamp() }}) as run_schedule_end_at,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by source_relation, id order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/stg_linkedin_ads__creative_history.sql

@@ -29,17 +29,21 @@ with base as (
         id as creative_id,
         campaign_id,
         coalesce(status, intended_status) as status,
-        click_uri,
+        coalesce(text_ad_landing_page, spotlight_landing_page, click_uri) as click_uri,
         cast(coalesce(last_modified_time, last_modified_at) as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(coalesce(created_time, created_at) as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by source_relation, id order by coalesce(last_modified_time, last_modified_at) desc) = 1 as is_latest_version
+        case when text_ad_landing_page is not null then 'text_ad'
+            when spotlight_landing_page is not null then 'spotlight'
+            else cast(null as {{ dbt.type_string() }})
+        end as click_uri_type
 
     from macro
 
 ), url_fields as (
 
     select 
         *,
+        row_number() over (partition by creative_id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_at desc) = 1 as is_latest_version,
         {{ dbt.split_part('click_uri', "'?'", 1) }} as base_url,
         {{ dbt_utils.get_url_host('click_uri') }} as url_host,
         '/' || {{ dbt_utils.get_url_path('click_uri') }} as url_path,