api updates as of 2/3 #48

fivetran-sheringuyen · 2023-02-06T18:53:15Z

❗ ❗
We are currently waiting for data to verify the updates before merging and releasing the changes in this PR. If you are currently using our Linkedin Source Package and would like to test out this version of the package, we would love to hear about your experience. Please feel free to comment below and let us know what your experience is!

For those interested, you can test out this version by adding the below in your packages.yml and temporarily comment out your current version of dbt_linkedin_source. Note, if you are using the dbt_linkedin package, please refer to the instructions for that package instead!

packages:
  - git: https://github.com/fivetran/dbt_linkedin_source.git
    revision: feature/linkedin-api-updates-2023
    warn-unpinned: false

❗ ❗
Are you a current Fivetran customer?

Fivetran Employee 💃

What change(s) does this PR introduce?

🚨 Breaking Changes 🚨

Due to Linkedin Ads API change in January 2023, there have been updates in the Linkedin Ads Fivetran Connector and therefore, updates to this Linkedin package.

The following fields have been completely deprecated in the stg_linkedin_ads__creative_history model (PR #48):

type
call_to_action_label_type
version_tag

Updates

The following legacy fields have been updated respectively in the connector and [PR api updates as of 2/3 #48](https://github.com/fivetran/ dbt_linkedin_source/pull/48) includes the below modifications:
- last_modified_time has been updated to last_modified_at
- created_time has been updated to created_at
- status has been updated to intended_status
src_linkedin.yml have been updated to reflect new definitions for the above updated fields.

Under the Hood

integration_tests/seeds/linkedin_creative_history_data has been updated to reflect new fields and deprecated fields
Uniqueness testing in stg_linkedin_ads__creative_history has been updated to now include _fivetran_synced and last_modified_at in place of the previous test with version_tag.

Did you update the CHANGELOG?

Yes

Does this PR introduce a breaking change?

Yes (please provide breaking change details below.)
No (please provide an explanation as to how the change is non-breaking below.)

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

Yes

Is this PR in response to a previously created Bug or Feature Request

Yes, Issue/Feature [link bug/feature number here]
No

How did you test the PR changes?

Buildkite
Local (please provide additional testing details below)

Select which warehouse(s) were used to test the PR

Provide an emoji that best describes your current mood

💃

Feedback

We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your feedback on our existing dbt packages or what you'd like to see next.

fivetran-sheringuyen · 2023-02-06T19:06:34Z

models/stg_linkedin.yml

@@ -61,7 +61,7 @@ models:
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
-            - version_tag
+            - _fivetran_synced


Previously, this was the version_tag column that seems to be deprecated - with no alternative solutions (for now). This should be just the last_modified_at/last_modified_timestamp column, however, there are currently duplicate records.

Coming back to this comment, if I recall you mentioned there was a rollout on the connector side that should have deleted these duplicate records. These tests should catch and raise duplicates, so aren't we intentionally skipping those with the addition of the _fivetran_synced field?

My proposal would be to either remove _fivetran_synced and catch the true duplicates so customers may be aware and address them, or remove the test altogether. What are your thoughts?

So I took some time to look into this a bit more today - in our data, it looks like there could still be duplicates if the creative object was deleted from the source and therefore no longer synced into the destination.

I don't think engineering plans on removing duplicates in the above instances and therefore even if the test captured duplicates, I'm not sure if there would be an action item. 🤔 While it doesn't 100% feel right for me to just remove the test all together, I don't know what other options we truly have here.

Removed and added messaging within the CHANGELOG!

fivetran-sheringuyen · 2023-02-06T19:42:21Z

To-do:

Docs to be generated before merge & release!

fivetran-joemarkiewicz

@fivetran-sheringuyen The changes look good; however, I will hold off on approving until we get confirmation from the product team that these are good to roll out.

Additionally, I had a super minor comment to address in the meantime.

dbt_project.yml

Csgoodman · 2023-03-03T23:54:57Z

models/src_linkedin.yml


          - name: campaign_id
            description: The ID of the campaign the creative belongs to.

-          - name: status
+          - name: click_uri


@fivetran-sheringuyen so that you are aware - the click_uri field will contain data for both text and spotlight ad types. No other ad types contain data for this.

@Csgoodman do you have an example of the structure of this updated field?

circling back for documentation - it seems that a creative can only be of a single type -- dynamic ad, text or reference to InMail Content. So this field will follow essentially the same formatting and will only hold data for one of the previously mentioned ads (no need to worry about comma separated lists or anything!)

fivetran-joemarkiewicz

@fivetran-sheringuyen thanks for working so hard to get through these API updates and ensure the changes to the package are properly reflected 🙏

I did have a few suggestions and a comment around the test on the staging model. Happy to chat more about this in person if you would prefer.

fivetran-joemarkiewicz · 2023-03-06T15:28:26Z

CHANGELOG.md

+- `version_tag`
+
+## Updates
+- The following legacy fields have been updated respectively in the connector and [PR #48](https://github.com/fivetran/ dbt_linkedin_source/pull/48) includes the below modifications:


This markdown link seems to be malformed. I think it is the extra space that may be trowing it off.

fivetran-joemarkiewicz · 2023-03-06T15:34:26Z

models/stg_linkedin.yml

@@ -61,7 +61,7 @@ models:
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
-            - version_tag
+            - _fivetran_synced


Coming back to this comment, if I recall you mentioned there was a rollout on the connector side that should have deleted these duplicate records. These tests should catch and raise duplicates, so aren't we intentionally skipping those with the addition of the _fivetran_synced field?

My proposal would be to either remove _fivetran_synced and catch the true duplicates so customers may be aware and address them, or remove the test altogether. What are your thoughts?

fivetran-joemarkiewicz · 2023-03-06T15:35:08Z

CHANGELOG.md

+
+## Under the Hood
+- `integration_tests/seeds/linkedin_creative_history_data` has been updated to reflect new fields and deprecated fields
+- Uniqueness testing in `stg_linkedin_ads__creative_history` has been updated to now include `_fivetran_synced` and `last_modified_at` in place of the previous test with `version_tag`. 


Same comment from the staging model about the inclusion of _fivetran_synced being included in the test. I would prefer we remove it if you think that is a possibility.

fivetran-joemarkiewicz

LGTM! Thanks for seeing this through to the end and working on addressing the necessary updates!

fivetran-sheringuyen added 2 commits February 3, 2023 15:22

api updates as of 2/3

c3996af

updating creative history tests

1b4093d

fivetran-sheringuyen commented Feb 6, 2023

View reviewed changes

modifying tests

22c1834

updating changelog

b800879

fivetran-sheringuyen mentioned this pull request Feb 6, 2023

Feature/linkedin api updates 2023 fivetran/dbt_linkedin#28

Merged

14 tasks

fivetran-joemarkiewicz self-requested a review February 7, 2023 16:22

fivetran-joemarkiewicz assigned fivetran-sheringuyen Feb 7, 2023

fivetran-joemarkiewicz requested changes Feb 7, 2023

View reviewed changes

dbt_project.yml Show resolved Hide resolved

fivetran-sheringuyen added 3 commits February 7, 2023 10:06

updating readme dependency

31279d4

coalesce intended status in creative history

4ff43b7

updating docs

878e97a

Csgoodman reviewed Mar 3, 2023

View reviewed changes

fivetran-joemarkiewicz requested changes Mar 6, 2023

View reviewed changes

fivetran-sheringuyen added 6 commits March 16, 2023 12:37

removing creative history test

76d428f

updating changelog

89cb26f

updating changelog and seed data

933a942

docs

b30ded8

updating databricks profile

4c284e8

updating integration test schema

ee6f2b6

fivetran-joemarkiewicz approved these changes Mar 22, 2023

View reviewed changes

updating changelog

7c6eca8

fivetran-sheringuyen merged commit 20e9775 into main Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api updates as of 2/3 #48

api updates as of 2/3 #48

fivetran-sheringuyen commented Feb 6, 2023 •

edited by fivetran-joemarkiewicz

Loading

fivetran-sheringuyen Feb 6, 2023 •

edited

Loading

fivetran-joemarkiewicz Mar 6, 2023

fivetran-sheringuyen Mar 11, 2023

fivetran-sheringuyen Mar 20, 2023

fivetran-sheringuyen commented Feb 6, 2023 •

edited

Loading

fivetran-joemarkiewicz left a comment

Csgoodman Mar 3, 2023 •

edited

Loading

fivetran-sheringuyen Mar 6, 2023

fivetran-sheringuyen Mar 10, 2023

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Mar 6, 2023

fivetran-joemarkiewicz Mar 6, 2023

fivetran-joemarkiewicz Mar 6, 2023

fivetran-sheringuyen Mar 20, 2023

fivetran-joemarkiewicz left a comment

api updates as of 2/3 #48

api updates as of 2/3 #48

Conversation

fivetran-sheringuyen commented Feb 6, 2023 • edited by fivetran-joemarkiewicz Loading

🚨 Breaking Changes 🚨

Updates

Under the Hood

fivetran-sheringuyen Feb 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-sheringuyen commented Feb 6, 2023 • edited Loading

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Csgoodman Mar 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

fivetran-sheringuyen commented Feb 6, 2023 •

edited by fivetran-joemarkiewicz

Loading

fivetran-sheringuyen Feb 6, 2023 •

edited

Loading

fivetran-sheringuyen commented Feb 6, 2023 •

edited

Loading

Csgoodman Mar 3, 2023 •

edited

Loading