Dashboard crawls should support date range as a parameter

[adborden]

Dashboard crawls depend on point-in-time versions of agency data.json files. If the jobs does not run on a specific day, there is no way to go back in time to get the data.json. This means that if a nightly job fails, it means data loss. If it fails several days in a row, it becomes a high priority issue.

Having a time sensitive operation is difficult for O&M.

How to reproduce

1. Disable the cron for day
2. Nightly cron does not run
3. Run the cron job the following day

Expected behavior

Crawl can create data for the missed day.

Actual behavior

Data is lost for the missed day.

[adborden]

Not sure that we can ever fix this. We don't have a time machine and we don't control agency data.json files. Still, we can document this undesired property and identify mitigations that make it less painful for O&M.

E.g. if the daily snapshotting of data.json became commoditized, the nightly job could be parameterized with a date range, and just operate on the snapshots.

[nickumia-reisys]

• Deprecate the Dashboard #3502