Skip to content

Latest commit

 

History

History
164 lines (114 loc) · 8.57 KB

secret-key-rotation-steps.md

File metadata and controls

164 lines (114 loc) · 8.57 KB
Raw

Rotating Secret Keys

Context

To maintain good security, we will periodically rotate the following secret keys, which are used to control authentication and authorization to our application:

  • CF deployer keys (for continuous delivery)
  • JWT keys (external user auth)
  • ACF AMS keys (internal user auth)
  • Django secret keys (cryptographic signing)

This document outlines the process for doing this for each set of keys.

Recommendation: Create a new issue to track key rotation for team awareness.

Warning(s):

  • Production sites will need to be taken down for maintenance when rotating keys, as the rotation will automatically invalidate all current sessions.
  • As of June 2022, CircleCI supplies environment variable key-value pairs to multiple environments (e.g. Raft's CircleCI deploys applications to dev and staging environments). The values from CircleCI are expected to be unique per environment, so until #1826 is researched and addressed, these values will need to be manually corrected in cloud.gov immediately following the execution of the <env>-deployment CircleCI workflow. This workaround applies to backend applications in the TDP staging environment.

Rotation procedures

CF deployer keys There are unique cloud foundry (CF) credentials for each cloud.gov space (tanf-dev, tanf-staging, tanf-prod) for deployments. These are stored in the tanf-keys service instance in cloud.gov. The steps below should be followed to rotate the credentials for the relevant space(s). Note: <ENV> := DEV, STAGING, or PROD.

  1. verify existing credentials for deployer key in tanf-keys service instance (these are the values for CF_USERNAME_<ENV> and CF_PASSWORD_<ENV> in circleci project settings
# target env space
cf target -o hhs-acf-ofa -s tanf-<env>

# verify deployment credentials
cf service-key tanf-keys deployer
  1. remove the current username associated the the deployer tanf-keys service instance from cloud.gov space (this is the same value as CF_USERNAME_<env> ). This task can also be done from the dashboard.
cf delete-user <<insert USERNAME value for deployer key>>
  1. delete the deployer service key associated with tanf-keys service instance (reference)
# delete
cf delete-service-key tanf-keys deployer

# verify deletion
cf service-keys tanf-keys
  1. create new deployer service key within tanf-keys instance (reference link above)
cf create-service-key tanf-keys deployer
  1. add username for newly generated deployer service key to space as a user to relevant space and assign as an org user and space developer. This task can also be done from the dashboard.
# add user
cf create-user <<insert USERNAME value for deployer key>>

# add as a user to org
cf set-org-role <<insert USERNAME value for deployer key>> hhs-acf-ofa OrgUser

# add as developer to prod space
cf set-space-role <<insert USERNAME value for deployer key>> hhs-acf-ofa tanf-prod SpaceDeveloper
  1. Confirm that the new deployment credentials work in CircleCI (re-run deployment workflow after adding CF_USERNAME_<env> and CF_PASSWORD_<env> back to CircleCI with rotated values)

JWT Keys

The following steps are applicable for lower environments (dev and staging) only. See here for prod environment procedure.

1. Generate New Keys

In your Mac terminal (or bash terminal in Windows), enter the following command:

yes 'XX' | openssl req -nodes -x509 -days 100 -newkey rsa:4096 -keyout jwtRS256prv.pem -out jwtRS256pub.crt

You can now check the contents of your keys with these commands

cat jwtRS256prv.pem
# returns private key
cat jwtRS256pub.crt
# returns public key

2. Base64 Encode Private Key

We use Base64 Encoded Private Keys to make it easier to save to cloud environments and local .env files.

openssl enc -base64 -in jwtRS256prv.pem -out jwtRS256prv.pem.base64

cat jwtRS256prv.pem.base64

NOTE: Linux users must disable line wrapping by adding the argument -w 0 to get a properly formatted one-line value.

cat jwtRS256prv.pem | base64 -w 0 > jwtRS256prv.pem.base64
cat jwtRS256prv.pem.base64

3. Copy Keys

Dev Environments

  1. Distribute the private key to development staff securely to copy to .env files as the value for key JWT_KEY
  2. Update the environment variables JWT_KEY with the base64-encoded private key and JWT_CERT in cloud.gov backend development and staging environments
cf set-env $cgbackendappname JWT_KEY $JWT_KEY_VALUE
cf set-env $cgbackendappname JWT_CERT "$JWT_CERT_VALUE\
> _IS_TYPICALLY_\
>MULTILINE"
  1. Login to the Login.gov Sandbox and verify the values are updated across all environments (4 dev + 2 staging)

Note: Login.gov requires the key to be uploaded in PEM format, which is the format we produced in the jwtRS256pub.crt file.

pem_upload

Staging Environments

Note Please generate a separate set of keys for the staging environments. The steps here will be the same as development but you will need to generate a separate key-value pair and upload them to the separate app listing in Login.gov's dashboard as linked above.

CI/CD Environment

  1. Distribute the private key to development staff securely to copy to .env
  2. Update the variables JWT_KEY and JWT_CERT_TEST in CircleCI with the new keypair.

Production Environment

Note: Please generate a separate set of keys for the Production environment.The steps here will be the same as development but you will need to generate a separate key-value pair and upload them to the separate app listing in Login.gov's dashboard as linked above.

Production environment key generation, change requests, and distribution will be handled by Government-authorized personnel with Government computers and PIV access (e.g. TDP sys admins)

  1. Copy the private key to cloud.gov backend environment variable JWT_KEY
  2. Copy the public key to the login.gov production environment
  3. In order for the key change to take effect, a change request must be submitted to the login.gov support portal. These requests can take approx. 2 weeks to be completed.

References

  • More information on openssl can be found at openssl.org

ACF AMS Keys The ACF OCIO Ops team manages these credentials for all environments (dev, staging, and prod), so we will need to submit a service request ticket whenever we need keys rotated.

Service requests tickets must be submitted by Government-authorized personnel with Government computers and PIV access (e.g. Raft tech lead for lower environments and TDP sys admins for production environment). Please follow the procedures below:

  1. Submit request tickets from government-issued email address and use the email template located on page 2 of this document. cc OFA tech lead on lower environment requests.
  2. Update environment variables in CircleCI and relevant cloud.gov backend applications after ticket completed by OCIO. Restage applications.

Django secret keys

DJANGO_SECRET_KEY is dynamically generated since #1151, so all that needs to be done to rotate this key in any environment is to re-run the relevant environment's deployment workflow in CircleCI. These are as follows:

  • dev environment workflow (dev-deployment) is run from CircleCI for raft-tech/TANF-app.
  • staging environment workflow (staging-deployment) is run from CircleCI for raft-tech/TANF-app via deploy-develop.
  • staging environment workflow (staging-deployment) is run from CircleCI for HHS/TANF-app via deploy-staging.
  • prod environment workflow (production-deployment) is run from CircleCI for HHS/TANF-app.