Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter out deleted accounts #57

Merged
merged 10 commits into from
Oct 30, 2024
Merged

Conversation

fivetran-avinash
Copy link
Contributor

PR Overview

This PR will address the following Issue/Feature: [#55]

This PR will result in the following new package version: 11.0

As deleted records are being filtered out here and in dbt_quickbooks, this changes the reporting and warrants a breaking change update.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes

  • We are currently bringing in records that have previously been deleted, which could be impacting reporting accuracy. Thus, we introduced the _fivetran_deleted field to filter out these records from the following staging models:
    • stg_quickbooks__account
    • stg_quickbooks__bundle
    • stg_quickbooks__customer
    • stg_quickbooks__department
    • stg_quickbooks__item
    • stg_quickbooks__vendor
  • As this will filter out deleted records that were previously being counted and we are adding a new field to these staging models, this will be a breaking change here and in dbt_quickbooks.

Documentation Updates

  • Added the _fivetran_deleted field to src_quickbooks and stg_quickbook yml documentation.
  • Added the _fivetran_deleted field to the above corresponding seed files in integration tests.

Contibutors

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • [NA] dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

Models ran and passed, and the new filters did take out deleted records.

If you had to summarize this PR in an emoji, which would it be?

🧢

@fivetran-avinash fivetran-avinash self-assigned this Oct 28, 2024
@fivetran-avinash fivetran-avinash linked an issue Oct 28, 2024 that may be closed by this pull request
4 tasks
Copy link
Contributor

@fivetran-catfritz fivetran-catfritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-avinash This generally looks good. Just need some additional documentation update and then a docs regen!

Comment on lines +51 to +52
- name: _fivetran_deleted
description: "{{ doc('_fivetran_deleted') }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definition needs to be added to the other staging models:

  • stg_quickbooks__bundle
  • stg_quickbooks__customer
  • stg_quickbooks__department
  • stg_quickbooks__item
  • stg_quickbooks__vendor

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

Copy link
Contributor Author

@fivetran-avinash fivetran-avinash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-catfritz This is ready for re-review!

Copy link
Contributor

@fivetran-catfritz fivetran-catfritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-avinash updates look good. Approved!

Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fivetran-avinash fivetran-avinash merged commit c2c1703 into main Oct 30, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Filter out deleted Accounts
4 participants