Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update field selection, etc #40

Merged

Conversation

fivetran-jamie
Copy link
Contributor

Are you a current Fivetran customer?

Fivetran made PR

What change(s) does this PR introduce?

  • incorporates new passthrough column format (will be documented in readme later)
  • temporarily points to a branch of fivetran utils where the passthrough column macros i have added are now backwards compatible
  • break out staging column macros into their own respective files
  • includes new fields (will document later)
  • moves aliasing of fields to staging models, where we no longer select *
  • renamed the CTEs to be consistent with new template
  • create an empty refund table if there's no source table (adapted from https://github.com/fivetran/dbt_zendesk_source/blob/main/models/tmp/stg_zendesk__ticket_schedule_tmp.sql)
  • reverts aliasing of _at fields to _timestamp (so we don't change their names. did this for consistency across the package)
  • removed a field that will be replaced by a new staging model

Did you update the CHANGELOG?

  • Yes

NOT ENTIRELY i just noted the models i switched the timestamp column names in bc it was a low lift

Does this PR introduce a breaking change?

  • Yes (please provide breaking change details below.)
  • No (please provide an explanation as to how the change is non-breaking below.)

its base branch is a breaking change

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

  • Yes

NA

Is this PR in response to a previously created Bug or Feature Request

How did you test the PR changes?

  • CircleCi
  • Local (please provide additional testing details below)

Select which warehouse(s) were used to test the PR

  • BigQuery
  • Redshift
  • Snowflake
  • Postgres
  • Databricks
  • Other (provide details below)

Provide an emoji that best describes your current mood

⛈️

Feedback

We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your feedback on our existing dbt packages or what you'd like to see next.

Copy link
Contributor

@fivetran-reneeli fivetran-reneeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Jamie! Just had some minor comments

models/stg_shopify.yml Show resolved Hide resolved
@@ -1,6 +1,15 @@
--To disable this model, set the shopify__using_refund variable within your dbt_project.yml file to False.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind explaining why this model is different than the rest?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i remember correctly, a request to do this came from a customer to @fivetran-joemarkiewicz ... joe would you mind elaborating here if i didn't make that up lol

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refund tables are only synced once a refund is provided. Therefore, a customer could go months without refunding anything and never use that table. However, the second a refund is provided this table will be synced.

The experience from the customer was that it was difficult to maintain when refunds started. This is not a "I don't use this feature" as it is always used, but will only start syncing once a refund is provided. Customers shouldn't have to worry about managing their dbt project once refunds are processed. Therefore we took the approach of having this refund table be one that syncs as empty if it is not populated, then seamlessly switches to the source table once it exists.

Copy link
Contributor

@fivetran-reneeli fivetran-reneeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@fivetran-jamie fivetran-jamie merged commit 6e2c05b into feature/package-revamp Dec 14, 2022
@fivetran-jamie fivetran-jamie deleted the feature/revamp/update-field-selection branch December 14, 2022 20:23
@fivetran-jamie fivetran-jamie mentioned this pull request Jan 31, 2023
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants