Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support uploading the exported-from-Postgres sqlite DB to cloud storage #14660

Closed
3 tasks
Tracked by #14225
stevejalim opened this issue Jun 10, 2024 · 1 comment
Closed
3 tasks
Tracked by #14225
Assignees
Labels
Backend Server stuff yo WIP 🚧 Pull request is still work in progress

Comments

@stevejalim
Copy link
Collaborator

stevejalim commented Jun 10, 2024

Description

Currently, the clock pod updates the DB and then uploads it to the cloud bucket for the other pods to pull down.

When we switch to postgres as Bedrock's DB backend, we won't have a sqlite DB to copy over to the bucket, but we do want to retain that overall behaviour, because then Bedrock can be used locally (or in things like www-sitemap-generator) without needing postgres.

In #14993 we improved our tooling to generate a sqlite DB from postgres. Now we need to run that exporter on a schedule and copy the resulting DB up to the regular cloud bucket that the clock pod used to do.

Ideally we should re-use any existing scripts so that it's basically just the source of bedrock.db that changes. When postgres is in use, the bedrock.db file is redundant, but our upload script knows to look for it and how to generate the JSON metadata file that goes with it, so it makes sense to try to repurpose as much of our tried-and-tested scripting as we can.

Note: this work will actually take place over in our (private) infra repo


Success Criteria

  • A fresh DB is generated, with appropriately scrubbed data
  • That DB is uploaded to the appropriate bucket
  • That DB can be pulled down and loaded in for use in any Bedrock container, as now
@alexgibson alexgibson added the Backend Server stuff yo label Jun 24, 2024
@stevejalim stevejalim self-assigned this Aug 27, 2024
@stevejalim stevejalim added the WIP 🚧 Pull request is still work in progress label Aug 29, 2024
@stevejalim stevejalim changed the title Update the database-upload script to generate a new DB export (from postgres) Support uploading the exported-from-Postgres sqlite DB to cloud storage Aug 29, 2024
@stevejalim
Copy link
Collaborator Author

This appears to be working

Screenshot 2024-09-03 at 23 28 29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Backend Server stuff yo WIP 🚧 Pull request is still work in progress
Projects
None yet
Development

No branches or pull requests

2 participants