Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the detector_config_log freshness check less sensitive #436

Merged
merged 5 commits into from
Jan 10, 2025
Merged

Make the detector_config_log freshness check less sensitive #436

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c7ea303
Update _sources.yml
pingpingxiu-DOT-ca-gov Oct 21, 2024
103f24e
Update _sources.yml
pingpingxiu-DOT-ca-gov Oct 21, 2024
1d5553e
Update _sources.yml
pingpingxiu-DOT-ca-gov Oct 21, 2024
2ef2cca
Fix trailing
pingpingxiu-DOT-ca-gov Oct 21, 2024
06f0920
Adjust freshness check values based on rough update frequencies of these
ian-r-rose Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 24 additions & 11 deletions transform/models/_sources.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,14 +169,6 @@ sources:
description: |
Vehicle Detector Station (VDS) (aka Station) related data obtained via data relay from
db96 within the Caltrans network.
freshness:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't really make sense to have this config at the schema level, as not every table has a time_id column. Removing this allows us to remove the null overrides below, and we can just add more specific tests on the tables we actually want to check

warn_after:
count: 2
period: day
error_after:
count: 3
period: day
loaded_at_field: time_id
tables:
- name: vds30sec
description: |
Expand Down Expand Up @@ -288,7 +280,6 @@ sources:
stations hooked up to a single controller. For example, the same
controller might have a station for the mainline as well as a station
for an onramp.
freshness: null
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand All @@ -310,6 +301,14 @@ sources:
description: |
A log table showing updates to the controller config. This can be joined
with the `controller_config` table to get a full history of controller metadata.
freshness:
warn_after:
count: 30
period: day
error_after:
count: 60
period: day
loaded_at_field: time_id
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand Down Expand Up @@ -341,7 +340,6 @@ sources:
description: |
Metadata for a single VDS station. Multiple stations may be connected to
a single controller, and multiple detectors may be connected to a single station.
freshness: null
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand All @@ -363,6 +361,14 @@ sources:
description: |
A log table showing updates to the station config. This can be joined
with the `station_config` table to get a full history of station metadata.
freshness:
warn_after:
count: 15
period: day
error_after:
count: 30
period: day
loaded_at_field: time_id
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand Down Expand Up @@ -400,7 +406,6 @@ sources:
Metadata for a single loop detector. This is the device that actually records
flow, occupancy, and speed, and is typically installed in a single lane.
Multiple detectors across a set of lanes constitute a station.
freshness: null
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand All @@ -412,6 +417,14 @@ sources:
description: |
A log table showing updates to the detector config. This can be joined
with the `detector_config` table to get a full history of detector metadata.
freshness:
warn_after:
count: 15
period: day
error_after:
count: 30
period: day
loaded_at_field: time_id
columns:
- name: meta
description: Metadata from the data relay scripts.
Expand Down
Loading