You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the clock pod updates the DB and then uploads it to the cloud bucket for the other pods to pull down.
When we switch to postgres as Bedrock's DB backend, we won't have a sqlite DB to copy over to the bucket, but we do want to retain that overall behaviour, because then Bedrock can be used locally (or in things like www-sitemap-generator) without needing postgres.
In #14993 we improved our tooling to generate a sqlite DB from postgres. Now we need to run that exporter on a schedule and copy the resulting DB up to the regular cloud bucket that the clock pod used to do.
Ideally we should re-use any existing scripts so that it's basically just the source of bedrock.db that changes. When postgres is in use, the bedrock.db file is redundant, but our upload script knows to look for it and how to generate the JSON metadata file that goes with it, so it makes sense to try to repurpose as much of our tried-and-tested scripting as we can.
Note: this work will actually take place over in our (private) infra repo
Success Criteria
A fresh DB is generated, with appropriately scrubbed data
That DB is uploaded to the appropriate bucket
That DB can be pulled down and loaded in for use in any Bedrock container, as now
The text was updated successfully, but these errors were encountered:
stevejalim
changed the title
Update the database-upload script to generate a new DB export (from postgres)
Support uploading the exported-from-Postgres sqlite DB to cloud storage
Aug 29, 2024
Description
Currently, the clock pod updates the DB and then uploads it to the cloud bucket for the other pods to pull down.
When we switch to postgres as Bedrock's DB backend, we won't have a sqlite DB to copy over to the bucket, but we do want to retain that overall behaviour, because then Bedrock can be used locally (or in things like
www-sitemap-generator
) without needing postgres.In #14993 we improved our tooling to generate a sqlite DB from postgres. Now we need to run that exporter on a schedule and copy the resulting DB up to the regular cloud bucket that the clock pod used to do.
Ideally we should re-use any existing scripts so that it's basically just the source of
bedrock.db
that changes. When postgres is in use, thebedrock.db
file is redundant, but our upload script knows to look for it and how to generate the JSON metadata file that goes with it, so it makes sense to try to repurpose as much of our tried-and-tested scripting as we can.Note: this work will actually take place over in our (private) infra repo
Success Criteria
The text was updated successfully, but these errors were encountered: