Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api updates as of 2/3 #48

Merged
merged 14 commits into from
Mar 23, 2023
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# dbt_linkedin_source v0.7.0
## 🚨 Breaking Changes 🚨
Due to Linkedin Ads API [change in January 2023](https://learn.microsoft.com/en-us/linkedin/marketing/integrations/recent-changes?view=li-lms-2022-12#january-2023), there have been updates in the Linkedin Ads Fivetran Connector and therefore, updates to this Linkedin package.

The following fields have been completely deprecated in the `stg_linkedin_ads__creative_history` model:
- `type`
- `call_to_action_label_type`
- `version_tag`

## Updates
The following legacy fields have been updated respectively in the connector and [PR #](LINK HERE) includes the below modifications:
- `last_modified_time` has been updated to `last_modified_at`
- `created_time` has been updated to `created_at`
- `status` has been updated to `intended_status`

# dbt_linkedin_source v0.6.0

## 🚨 Breaking Changes 🚨:
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'linkedin_source'
version: '0.6.0'
version: '0.7.0'
fivetran-sheringuyen marked this conversation as resolved.
Show resolved Hide resolved
config-version: 2
require-dbt-version: [">=1.3.0", "<2.0.0"]
models:
Expand Down
4 changes: 3 additions & 1 deletion integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'linkedin_source_integration_tests'
version: '0.6.0'
version: '0.7.0'
profile: 'integration_tests'
config-version: 2

Expand All @@ -20,6 +20,8 @@ seeds:
+column_types:
created_time: timestamp
last_modified_time: timestamp
created_at: timestamp
last_modified_at: timestamp
click_uri: "{{ 'string' if target.type in ('bigquery','spark','databricks') else 'varchar' }}"
linkedin_ad_analytics_by_creative_data:
+column_types:
Expand Down
7 changes: 3 additions & 4 deletions macros/get_creative_history_columns.sql
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
{% macro get_creative_history_columns() %}

{% set columns = [
{"name": "call_to_action_label_type", "datatype": dbt.type_string()},
{"name": "campaign_id", "datatype": dbt.type_int()},
{"name": "click_uri", "datatype": dbt.type_string()},
{"name": "created_time", "datatype": dbt.type_timestamp()},
{"name": "created_at", "datatype": dbt.type_timestamp()},
{"name": "id", "datatype": dbt.type_int()},
{"name": "last_modified_time", "datatype": dbt.type_timestamp()},
{"name": "status", "datatype": dbt.type_string()},
{"name": "type", "datatype": dbt.type_string()},
{"name": "version_tag", "datatype": dbt.type_string()}
{"name": "last_modified_at", "datatype": dbt.type_timestamp()},
{"name": "intended_status", "datatype": dbt.type_string()},
] %}

{{ return(columns) }}
Expand Down
40 changes: 14 additions & 26 deletions models/src_linkedin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,45 +45,33 @@ sources:
description: Unique internal ID representing the creative.

- name: last_modified_time
description: The timestamp of when the version was updated.
description: The timestamp of when the version was updated (Linkedin Ads pre-January 2023 API Update).

- name: last_modified_at
description: The timestamp of when the version was updated (Linkedin Ads post-January 2023 API Update).

- name: created_time
description: The timestamp of when the creative was created.
description: The timestamp of when the creative was created (Linkedin Ads pre-January 2023 API Update).

- name: created_at
description: The timestamp of when the creative was created (Linkedin Ads post-January 2023 API Update).

- name: campaign_id
description: The ID of the campaign the creative belongs to.

- name: status
- name: click_uri
Copy link

@Csgoodman Csgoodman Mar 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-sheringuyen so that you are aware - the click_uri field will contain data for both text and spotlight ad types. No other ad types contain data for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Csgoodman do you have an example of the structure of this updated field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

circling back for documentation - it seems that a creative can only be of a single type -- dynamic ad, text or reference to InMail Content. So this field will follow essentially the same formatting and will only hold data for one of the previously mentioned ads (no need to worry about comma separated lists or anything!)

description: |
Required when creative type is TEXT_AD. Maximum character limit of 500 after URL encoding all special characters.
Limit does not include the "https://" prefix.

- name: intended_status
description: |
ACTIVE - Creative creation is complete and creative is available for review and serve.
PAUSED - Creative creation is complete and creative is current, but should temporarily not be served. This status is not valid upon creation.
DRAFT - Creative creation is incomplete and may still be edited.
ARCHIVED - Creative creation is complete, but creative should not be served and should be separated from non-archived creatives in any UI.
CANCELED - Creative is permanently canceled.

- name: version_tag
description: An integer representation of the version of the creative.

- name: type
description: |
TEXT_AD - Text with Sponsored Link to landing page on your site.
SPONSORED_STATUS_UPDATE - Sponsored status update from a profile page.
SPONSORED_INMAILS - Single message ad delivered to the member's inbox
SPONSORED_MESSAGE - Conversation ad containing a tree of pre-determined messages delivered to the member's inbox
SPONSORED_VIDEO - Sponsored videos.
SPONSORED_UPDATE_CAROUSEL - Sponsored Carousels
FOLLOW_COMPANY_V2 - Dynamic Follow Ad
SPOTLIGHT_V2 - Dynamic Spotlight Ad
JOBS_V2 - Dynamic Job Ad

- name: click_uri
description: |
Required when creative type is TEXT_AD. Maximum character limit of 500 after URL encoding all special characters.
Limit does not include the "https://" prefix.

- name: call_to_action_label_type
description: The labeled call-to-action on this ad.

- name: campaign_history
description: Each record represents a 'version' of a campaign.
identifier: "{{ var('linkedin_ads_campaign_history_identifier', 'campaign_history') }}"
Expand Down
22 changes: 1 addition & 21 deletions models/stg_linkedin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ models:
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- version_tag
- _fivetran_synced
Copy link
Contributor Author

@fivetran-sheringuyen fivetran-sheringuyen Feb 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, this was the version_tag column that seems to be deprecated - with no alternative solutions (for now). This should be just the last_modified_at/last_modified_timestamp column, however, there are currently duplicate records.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to this comment, if I recall you mentioned there was a rollout on the connector side that should have deleted these duplicate records. These tests should catch and raise duplicates, so aren't we intentionally skipping those with the addition of the _fivetran_synced field?

My proposal would be to either remove _fivetran_synced and catch the true duplicates so customers may be aware and address them, or remove the test altogether. What are your thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I took some time to look into this a bit more today - in our data, it looks like there could still be duplicates if the creative object was deleted from the source and therefore no longer synced into the destination.

I don't think engineering plans on removing duplicates in the above instances and therefore even if the test captured duplicates, I'm not sure if there would be an action item. 🤔 While it doesn't 100% feel right for me to just remove the test all together, I don't know what other options we truly have here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed and added messaging within the CHANGELOG!

- creative_id
columns:
- name: creative_id
Expand All @@ -78,23 +78,6 @@ models:
- name: campaign_id
description: The ID of the campaign the creative belongs to.

- name: type
description: |
TEXT_AD - Text with Sponsored Link to landing page on your site.
SPONSORED_STATUS_UPDATE - Sponsored status update from a profile page.
SPONSORED_INMAILS - Single message ad delivered to the member's inbox
SPONSORED_MESSAGE - Conversation ad containing a tree of pre-determined messages delivered to the member's inbox
SPONSORED_VIDEO - Sponsored videos.
SPONSORED_UPDATE_CAROUSEL - Sponsored Carousels
FOLLOW_COMPANY_V2 - Dynamic Follow Ad
SPOTLIGHT_V2 - Dynamic Spotlight Ad
JOBS_V2 - Dynamic Job Ad

- name: version_tag
description: An integer representation of the version of the creative.
tests:
- not_null

- name: status
description: |
ACTIVE - Creative creation is complete and creative is available for review and serve.
Expand Down Expand Up @@ -132,9 +115,6 @@ models:
- name: utm_term
description: The utm_term parameter of the ad, extracted from the `click_uri`.

- name: call_to_action_label_type
description: The labeled call-to-action on this ad.

- name: is_latest_version
description: Boolean of whether the record is the latest version of the cretive.

Expand Down
11 changes: 4 additions & 7 deletions models/stg_linkedin_ads__creative_history.sql
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,11 @@ with base as (
select
id as creative_id,
campaign_id,
type,
cast(version_tag as numeric) as version_tag,
status,
intended_status as status,
click_uri,
call_to_action_label_type,
cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
row_number() over (partition by id order by last_modified_time desc) = 1 as is_latest_version
cast(coalesce(last_modified_at, last_modified_time) as {{ dbt.type_timestamp() }}) as last_modified_at,
cast(coalesce(created_at, created_time) as {{ dbt.type_timestamp() }}) as created_at,
row_number() over (partition by id order by coalesce(last_modified_at, last_modified_time) desc) = 1 as is_latest_version

from macro

Expand Down