diff --git a/backend/Makefile b/backend/Makefile index 9e0ab45d36296..07e486b7f4d02 100644 --- a/backend/Makefile +++ b/backend/Makefile @@ -63,17 +63,19 @@ db/dump: PGPASSWORD=${DB_PW} pg_dump -Fc --dbname=corpora_${DEPLOYMENT_STAGE} --file=$(OUTFILE) --host 0.0.0.0 --username corpora_${DEPLOYMENT_STAGE} $(MAKE) db/tunnel/down -db/load/local: +db/local/load-data: # Loads corpora_dev.sqlc into the local Docker env corpora database - # Usage: make db/load/local INFILE= - docker-compose exec database pg_restore --clean --no-owner --username corpora --dbname corpora $(INFILE) + # Usage: make db/local/load-data INFILE= + $(eval DB_PW = $(shell aws secretsmanager get-secret-value --secret-id corpora/backend/test/database --region us-west-2 | jq -r '.SecretString | match(":([^:]*)@").captures[0].string')) + PGPASSWORD=${DB_PW} pg_restore --clean --no-owner --host 0.0.0.0 --username corpora --dbname corpora $(INFILE) -db/load/schema: +db/local/load-schema: # Imports the corpora_dev.sqlc schema (schema ONLY) into the corpora_test database - # Usage: DEPLOYMENT_STAGE=test make db/import/schema - pg_restore --schema-only --clean --no-owner --dbname corpora_test corpora_$(DEPLOYMENT_STAGE).sqlc + # Usage: make db/local/load-schema INFILE= + $(eval DB_PW = $(shell aws secretsmanager get-secret-value --secret-id corpora/backend/test/database --region us-west-2 | jq -r '.SecretString | match(":([^:]*)@").captures[0].string')) + PGPASSWORD=${DB_PW} pg_restore --schema-only --clean --no-owner --host 0.0.0.0 --dbname corpora $(INFILE) # Also import alembic schema version - pg_restore --data-only --table=alembic_version --no-owner --dbname corpora_test corpora_$(DEPLOYMENT_STAGE).sqlc + PGPASSWORD=${DB_PW} pg_restore --data-only --table=alembic_version --no-owner --host 0.0.0.0 --dbname corpora $(INFILE) db/dump_schema: ifeq ($(DEPLOYMENT_STAGE),test) diff --git a/backend/database/README.md b/backend/database/README.md index ad8014836bad2..5eaacb0342efe 100644 --- a/backend/database/README.md +++ b/backend/database/README.md @@ -2,8 +2,8 @@ - General information about Alembic migrations can be found [here](https://alembic.sqlalchemy.org/en/latest/index.html). - For database [make recipes](../Makefile) -- See [Environment variables](../../README.md#environment-variables) for -usage explanation of `DEPLOYMENT_STAGE`, `AWS_PROFILE` and `CORPORA_LOCAL_DEV` +- See [Environment variables](../../README.md#environment-variables) for + usage explanation of `DEPLOYMENT_STAGE`, `AWS_PROFILE` and `CORPORA_LOCAL_DEV` - `$REPO_ROOT` - root directory where the `single-cell-data-portal` project is cloned (e.g. `~/PyCharmProjects/single-cell-data-portal`) ## How to perform a database migration @@ -14,13 +14,14 @@ usage explanation of `DEPLOYMENT_STAGE`, `AWS_PROFILE` and `CORPORA_LOCAL_DEV` 3. In the generated file, edit the `upgrade()` and `downgrade()` functions such that `upgrade()` contains the Alembic DDL commands to perform the migration you would like and `downgrade()` contains the commands to undo it. 4. Rename the generated file by prepending the migration count to the filename (`xxx_purpose_of_migration.py` -> `03_xxx_purpose_of_migration.py`) 5. In the generated file, update the `Revision ID` and the `revision` (used by Alembic) to include the migration count. -For example `Revision ID: a8cd0dc08805` becomes `Revision ID: 18_a8cd0dc08805` and `revision = "a8cd0dc08805"` becomes `revision = "18_a8cd0dc08805"` + For example `Revision ID: a8cd0dc08805` becomes `Revision ID: 18_a8cd0dc08805` and `revision = "a8cd0dc08805"` becomes `revision = "18_a8cd0dc08805"` 6. [Test your migration](#test-a-migration) 7. Check that [corpora_orm.py](../corpora/common/corpora_orm.py) matches up with your changes. 8. Once you've completed the changes, create a PR to get the functions reviewed. 9. Once the PR is merged, migrations will be run as part of the deployment process to each env. 10. [Connect to Remote RDS](#connect-to-remote-rds) to single-cell-dev 11. In a new terminal, complete the migration in the single-cell-dev test env by running: + ```shell cd $REPO_ROOT/backend DEPLOYMENT_STAGE=test make db/migrate @@ -30,55 +31,68 @@ DEPLOYMENT_STAGE=test make db/migrate 1. Make changes to the ORM class(es) in [corpora_orm.py](../corpora/common/corpora_orm.py) 2. [Connect to Remote RDS](#connect-to-remote-rds). Note, generally, you would be connecting to prod - (`AWS_PROFILE=single-cell-prod DEPLOYMENT_STAGE=prod`) since we want to generate -a migration from the database schema currently deployed in prod. However, if there are migrations haven't been -deployed to prod yet, you would connect to staging here. + (`AWS_PROFILE=single-cell-prod DEPLOYMENT_STAGE=prod`) since we want to generate + a migration from the database schema currently deployed in prod. However, if there are migrations haven't been + deployed to prod yet, you would connect to staging here. 3. Autogenerate the migration using the steps below. `AWS_PROFILE` and `DEPLOYMENT_STAGE` should be the same values -used in the previous [Connect to Remote RDS](#connect-to-remote-rds) step. For details about Alembic's migration autogeneration, -see [What does Autogenerate Detect (and what does it not detect?)](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect) + used in the previous [Connect to Remote RDS](#connect-to-remote-rds) step. For details about Alembic's migration autogeneration, + see [What does Autogenerate Detect (and what does it not detect?)](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect) + ```shell cd $REPO_ROOT/backend AWS_PROFILE=single-cell-{dev,prod} DEPLOYMENT_STAGE={dev,staging,prod} CORPORA_LOCAL_DEV=1 make db/new_migration_auto MESSAGE="purpose_of_migration" ``` -4. Follow [How to perform a database migration](#how-to-perform-a-database-migration) starting from **step 3** -(i.e. editing the `upgrade()` and `downgrade()` functions). + +4. Follow [How to perform a database migration](#how-to-perform-a-database-migration) starting from **step 3** + (i.e. editing the `upgrade()` and `downgrade()` functions). ### Test a Migration -The following steps will test that a migration script works on a local database using data downloaded from a deployed database. + +The following steps will test that a migration script works on a local database using data downloaded from a deployed database. 1. [Connect to Remote RDS](#connect-to-remote-rds) -2. Open a new terminal and using the same values for `AWS_PROFILE` and `DEPLOYMENT_STAGE`, download the remote database schema: +2. Open a new terminal and using the same values for `AWS_PROFILE` and `DEPLOYMENT_STAGE`, download the remote dev database schema: + ```shell cd $REPO_ROOT/backend -AWS_PROFILE=single-cell-{dev,prod} DEPLOYMENT_STAGE={dev,staging,prod} make db/download +AWS_PROFILE=single-cell-{dev,prod} DEPLOYMENT_STAGE={dev,staging,prod} make db/dump OUTFILE=corpora_dev.sqlc ``` -This will download the database into the `$REPO_ROOT/backend/database` directory in a file named `corpora_${DEPLOYMENT_STAGE}-.sqlc`. -3. Close the tunnel to the remote database +This will download the database to `$REPO_ROOT/backend/corpora_dev.sqlc`. + +3. The tunnel to dev should close automatically (but worth verifying `ps ax | grep ssh`) 4. Start the local database environment: + ```shell cd $REPO_ROOT make local-start ``` + 5. Import the remote database schema into your local database: + ```shell cd $REPO_ROOT/backend -DEPLOYMENT_STAGE=test make db/import FROM=corpora_{dev,staging,prod}- # exclude the .sqlc extension +make db/local/load-schema INFILE=corpora_dev.sqlc ``` -where the `FROM` parameter is the base name of the `.sqlc` file downloaded from the `make db/download` step above. For example + +where the `INFILE` parameter is the base name of the `.sqlc` file downloaded from the `make db/dump` step above. For example + ```shell -make db/import FROM=corpora_prod-202102221309 +make db/local/load-schema INFILE=corpora_dev.sqlc ``` -- Note: The file is stored locally under `$REPO_ROOT/backend/database/corpora_${DEPLOYMENT_STAGE}-.sqlc` -but the `make db/import` command retrieves it from `/import/$(FROM).sqlc` due to the way [the local paths are mapped to the Docker container](https://github.com/chanzuckerberg/single-cell-data-portal/blob/ffca067b9e4aea237fa2bd7c7a9cbc5813ebd449/docker-compose.yml#L13) You may need to run this a few times, until there are no significant errors. - - Note: `pg_restore: error: could not execute query: ERROR: role "rdsadmin" does not exist` is not a significant error + +- Note: `pg_restore: error: could not execute query: ERROR: role "rdsadmin" does not exist` is not a significant error + 6. Run the migration test: + ```shell AWS_PROFILE=single-cell-{dev,prod} DEPLOYMENT_STAGE=test make db/test_migration -``` +``` + This test will: + 1. Dump the current schema (before) 1. Apply the migration (upgrade) 1. Rollback the migration (downgrade) @@ -88,19 +102,18 @@ This test will: If there are no differences then the test passed. If the test didn't pass, make adjustments to your migration script and restart from step 5. Repeat until there are no errors. ## Connect to Remote RDS + Enable local connection to the private RDS instance: - Note: Since the default PostgreSQL port is `5432`, the above command will conflict with a local PostgreSQL instance. -To stop it run `make local-stop` from the `$REPO_ROOT` directory. - + To stop it run `make local-stop` from the `$REPO_ROOT` directory. ```shell cd $REPO_ROOT/backend AWS_PROFILE=single-cell-{dev,prod} DEPLOYMENT_STAGE={dev,staging,prod} make db/tunnel ``` -This command opens an SSH tunnel from `localhost:5432` to the RDS connection endpoint via the *bastion* server. -The local port `5432` is fixed and encoded in the DB connection string stored in +This command opens an SSH tunnel from `localhost:5432` to the RDS connection endpoint via the _bastion_ server. +The local port `5432` is fixed and encoded in the DB connection string stored in [AWS Secrets Manager](https://us-west-2.console.aws.amazon.com/secretsmanager/home?region=us-west-2#!/listSecrets/) in the secret named `corpora/backend/${DEPLOYMENT_STAGE}/database_local`. -