A service for tracking deployments (what's deployed and where) across environments and for discovering, persisting and providing access to all deployable artifacts on the platform (docker and ECR images).
TL;DR the CDP Portal Backend listens to SQS events for ECR image uploads and ECS deployments, populates a mongodb database with that information, and provides a RESTful API to access that information for the frontend.
Deployments - the "what is where"
cdp-portal-backend listens for AWS ECS deployment events on an SQS queue.
When an event comes in it will do the following:
- looks up the aws account id against a list of known environment names
- extracts the container image name and version from the event
- discards any containers that dont already exist
- writes the event to the deployments collection
Artifacts and images - what we can deploy on the portal
cdp-portal-backend listens for SQS events from the AWS ECR repository. When a new container is pushed, this service will download the manifest of the image.
If it hasn't seen the image before it will attempt to retrieve all the layers of the image, including the config layer. As it encounters files/data of interest it will read and
store this data.
Currently it extracts:
- Labels (specifically ones added during the build process to link it to the git repo that created it)
- Package.json (this is an example of how file extraction might work. In practice we'd want to search for package-lock.json etc)
For the most up-to-date APIs and schema, run cdp-portal-backend locally, as instructed below, and navigate to
the Swagger page at http://localhost:5094/swagger.
When an image is scanned its 'service name' is extracted from a docker label (defra.cdp.service.name see ArtifactScanner.cs).
GET /services
[
{
"serviceName": "cdp-portal-backend",
"githubUrl": "https://github.com/DEFRA/cdp-portal-backend",
"imageName": "cdp-portal-backend"
},
{
"serviceName": "cdp-portal-backend",
"githubUrl": "https://github.com/DEFRA/cdp-portal-backend",
"imageName": "cdp-portal-backend"
}
]
GET /services/cdp-portal-backend
{
"serviceName": "cdp-portal-backend",
"githubUrl": "https://github.com/DEFRA/cdp-portal-backend",
"imageName": "cdp-portal-backend"
}GET /artifacts
GET /artifacts/foo/1.2.3
{
"created":"2023-05-16T11:03:47.732Z",
"repo":"cdp-portal-backend",
"tag":"v0.21.0",
"sha256":"sha256:157e63cedba182003e9831047e3f611c516e8d67a4425dfcb15d7d7295c17872",
"githubUrl":"https://github.com/DEFRA/cdp-portal-backend",
"serviceName":"cdp-portal-backend",
"scannerVersion":1,
"files":[
{"fileName":"Defra.Cdp.Deployments.deps.json","path":"app/Defra.Cdp.Deployments.deps.json","layerSha256":"sha256:115ae824b21b9f671e304112f034096061d8d56b655509561207084fd1f3ccde"}
]
}GET /files/sha256:115ae824b21b9f671e304112f034096061d8d56b655509561207084fd1f3ccde?path=app%2FDefra.Cdp.Deployments.deps.json
(you'll likely need to url encode the path parameter)
{
"name": "cdp-node-frontend-exemplar",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
...Takes the layerSha256 value from the artifact data and the fileName as a query param. Returns a 404 if the filename is invalid. If you can retrieve the artifact, you have all the data required to access the file (digest and path).
GET /deployables
GET /deployables/image-name
Both provide a list of docker image names and tags for that image.
POST /admin/scan?repo=foo&tag=1.2.3
Triggers a scan of the given image/tag. This will grab the Docker image from ECR, scan the manifest and updates the artifacts collection.
POST /admin/backfill
Rescan everything in the docker registry. This will take a long time!
Currently it will not drop the existing records, that should be done manually.
GET /deployments returns the most recent deployment events, 1 per task. For services with more than one instance you'll get 1 event per instance rather than the whole deployment
GET /deployments/:deploymentId returns details about a specific deployment
GET /whats-running-where return the most recent deployment for each service in each environment
GET /whats-running-where/:service return the most recent deployment for a given service in each environment
GET /repositories returns all repositories for the organisation
GET /repositories?team=exampleTeam returns repositories owned by this team
GET /templates returns all repositories that are templates in the Portal
GET /templates?team=exampleTeam returns templates for a specific team
- Install MongoDB on your local machine
- Consider docker if running on Linux or WSL2 for Windows 10/11
- Consider Homebrew for MacOS as Mongodb in docker for Apple Silicon can be temperamental
sudo mongod --dbpath ~/mongodb-cdpNote: using the ubuntu image to avoid licensing issues with the ubi8 (redhat linux) image
docker run --name mongodb-cdp -d -p 27017:27017 mongodb/mongodb-community-server:6.0.7-ubuntu2204Note: if installing mongodb community edition
brew services start mongodb-community@6.0To inspect the Database and Collections locally:
mongosh- Set optional
ASPNETCORE_ENVIRONMENTenvironment variable:
export ASPNETCORE_ENVIRONMENT=Development
export GITHUB__TOKEN=<token_from_secret_manager>- To run the
cdp-portal-backendapplication:
dotnet run --project Defra.Cdp.Backend.Api --no-launch-profileIf you use Jetbrains Rider, make sure you create a configuration for .NET Project and not
.NET Launch Setting Profile so you can add your own environment variable there.
It should look something like this:
Running a local docker registry enables the cdp-portal-backend to obtain the manifests from local registry images.
docker run -d -p 5000:5000 --restart=always --name registry registry:2Note: If you need to update the external port also change it in the
Docker.RegistryUrlproperty withinappsettings.Development.json
You will then need to pull an image from ECR. This assumes you have AWS CLI setup correctly.
aws ecr get-login-password --region eu-west-2 --profile management | docker login --username AWS --password-stdin <account_number>.dkr.ecr.eu-west-2.amazonaws.com
docker pull <account_number>.dkr.ecr.eu-west-2.amazonaws.com/cdp-portal-frontend:<version>Then tag the pulled image
docker tag <account_number>.dkr.ecr.eu-west-2.amazonaws.com/cdp-portal-frontend:<version> localhost:5000/cdp-portal-frontend:<version>The push the tagged image to your local docker registry
docker push localhost:5000/cdp-portal-frontend:<version> This will make sure that the cdp-portal-backend can obtain the manifest for the versions in our local MongoDB.
- Install LocalStack AWS CLI
- Run AWS LocalStack Docker container:
docker run --pull=always -d -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack:latest- Create
ecs-deploymentslocal SQS queue:
awslocal sqs create-queue --queue-name ecs-deploymentsSend a message to the ecr-push-events queue, simulates a docker image available in the docker registry, for deployment.
This assumes:
- The AWS LocalStack Docker container is running and AWS LocalStack is installed.
- The appropriate docker image has been added to your local docker registry.
If you're not using AWS LocalStack, just replace the command with the normal aws command line + localstack connection details.
- Create the
ecr-push-eventsqueue:
awslocal sqs create-queue --queue-name ecr-push-events- Send an event:
awslocal sqs send-message --queue-url "http://127.0.0.1:4566/000000000000/ecr-push-events" --message-body '{"detail": { "result": "SUCCESS", "action-type": "PUSH", "image-tag": "0.1.0", "repository-name": "cdp-portal-frontend"}}'To Generate fake deployments across environments.
This assumes:
- The appropriate docker image has been added to your local docker registry.
- An
ecr-push-eventsSQS queue message has been sent
cd cdp-portal-backend./generate-fake-deployments.sh service-name versionE.g:
./generate-fake-deployments.sh cdp-portal-frontend 0.1.0