Skip to content

1.0.1 Refactor for daily+streaming & multiple Properties setup

Latest
Compare
Choose a tag to compare
@jice-lavocat jice-lavocat released this 22 Jun 14:37

52 Entertainment version of GA4 including 2 changes


Description & motivation

Merge PR between PR188 and PR231

PR188 - Refactor daily & streaming support

Previously, this package supported daily + streaming (aka intraday) exports in a relatively naïve way by incrementally processing 'daily' data and then unioning the 'streaming' data as a view in the stg_ga4__events model. This can cause issues when tables are rotated from 'intraday' to 'streaming' as noted in Velir#80 and in other cases noted in Velir#182.

This update ensures that daily & streaming data are handled using the same method: dats is loaded incrementally into the base model and deduped before being loaded into stg_ga4__events.

As a byproduct, this PR removes support for 'dynamically' incremental processing based on the last date found in a table using _max_partition. Instead, the static_incremental_days variable will need to be used which will ensure that data is reprocessed as the intraday table changes throughout the day.

PR231 - Implementing multi-project architecture ( issue Velir#231 )

Updating the dbt_projects.yml format to specify the GCP project and GA4 property_id.

Use case: when sources are spread on different distinct GCP Projects. Related to Velir#231

Note: This new config format can allow for a more specific configuration (example if property 11111111 is on daily frequency while property 22222222 is on streaming import this could be implemented in this way too in the future).