Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicitly truncate updated_at and created_at #132

Merged
merged 3 commits into from
Apr 22, 2024

Conversation

ericaporter
Copy link
Contributor

We need to explicitly truncate updated_at and created_at timestamps to milliseconds to be able to compare checksums between production db tables and Dataform

@ericaporter ericaporter requested a review from asatwal April 19, 2024 11:24
Copy link
Collaborator

@asatwal asatwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevenleggdfe @ericaporter

The where clause here:

def build_where_clause(order_column, table_name_sanitized, checksum_calculated_at_sanitized)

Which is used to filter out anything before checksum_calculated_at still uses microseconds precision for the order column. This will be different in BQ as it doesn't have microseconds. We just need to make this consistent also between dfe-analytics and BQ.

@stevenleggdfe
Copy link
Contributor

@stevenleggdfe @ericaporter

The where clause here:

def build_where_clause(order_column, table_name_sanitized, checksum_calculated_at_sanitized)

Which is used to filter out anything before checksum_calculated_at still uses microseconds precision for the order column. This will be different in BQ as it doesn't have microseconds. We just need to make this consistent also between dfe-analytics and BQ.

Good point. It's easy enough for me to change that in dfe-analytics-dataform if @ericaporter you are happy to change in dfe-analytics as well?

@ericaporter ericaporter force-pushed the fix-checksum-timestamps branch from 1b73148 to d66cd0d Compare April 22, 2024 10:02
Copy link
Collaborator

@asatwal asatwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM

@ericaporter ericaporter merged commit 7af1a0a into main Apr 22, 2024
7 checks passed
@ericaporter ericaporter deleted the fix-checksum-timestamps branch April 22, 2024 14:55
This was referenced Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants