Migrate build to GitHub actions #1151

DavidLawes · 2023-08-21T08:47:19Z

What does this change?

This PR adds GitHub workflows for the build of each of the apps inside in the n10n repo. Previously the builds were managed by TeamCity, but after this change is merged the corresponding TeamCity config will be deleted.

The current TeamCity configuration builds all projects, on every commit to every branch. The workflows in this PR change things slightly:

Each project has its own workflow file
Each project will be built on every commit, but only if the code changed in the commit matches one of the paths specified in the corresponding workflow
Instead of using sbt-riffraff-artifact, we now upload assets to riff-raff using the Guardian's action instead.

Each workflow follows one of these patterns:

Lambda assembly: event-consumer, fake-breaking-news, football, report-extractor, schedule-lambda, slo-monitor. This workflow typically just compiles and tests the project, then uses the sbt assembly command to generate a .jar.
Debian package: notification, registration, report. This workflow typically compiles and tests the project, then uses the debian:packageBin command (using plugin sbt-native-packager) to generate a .deb.
Docker image: notification-worker-lambdas. This workflow has to do something slightly more fiddly, it needs to upload images to ECR, which requires a valid authorisation token. There is some extra setup and steps in this type of workflow.

NB: not all projects have been converted to CDK, so the CDK synth step is not always required.

Docker lambdas

The notificationworkerlambda project deploys to docker lambdas, the historical reason for this is because the size of the .zip exceeded AWS's maximum size whereas docker lambdas don't impose such a limit.

To use docker lambdas we need to upload images to a remote ECR repository. In order to push images to the repository we need to login to ECR. Previously we managed the docker auth/deploy using script, in the new workflow I've changed things slightly:

I use an aws action to manage the login to ECR
I pulled out the docker commands into the workflow step explicitly (I think this helps the readability of the workflow file)

For a login to be successful we need to supply a custom role. I followed a couple of guides, which I'll link inline below, but my starting point was here. The custom role has been configured as following:

Create a new identity provider in the mobile aws account for the GitHub token provider service
Define a new role in the account that has a trust relationship with the GitHub identity provider, restricting access only to the n10n repo, following this guide.
Give the new role permissions to get an ECR token and put an ECR image (a bit of trial and error to work out the correct permissions)
Provide the role as a repository secret and use the role in the workflow.

I manually created the new role in the aws account, it's name is mobile-n10n-deploy-role. I believe I've applied the minimum permissions necessary for the workflow to succeed (including limiting the scope of any permissions applied to just the n10n ECR repository):

To get the token: ecr:GetAuthorizationToken
To publish an image: ecr:CompleteLayerUpload, ecr:UploadLayerPart, ecr:InitiateLayerUpload, ecr:BatchCheckLayerAvailability, ecr:PutImage

Given login was successful, the workflow compiles and tests the code and then uses the docker publishLocal command to generate a local image.

A small change I had to make was to explicitly tag the local image using the expected naming convention ($repository-url:$build-number). I'm not quite sure how this was working in the TeamCity build, but this current solution does work. If we migrated to using a docker build command, I think we can explicitly set a tag value, but I couldn't see anything obvious/immediate in the packager docs.

Given a local docker image exists with the expected tag we can then push the image to the remote ECR repository.

Testing

Service	Testing
Event consumer	Successful deploy to CODE, lambda continued to be triggered every 5 mins, no errors in logs, athena queries were successfully executed.
Fake breaking news lambda	Successful deploy to CODE, lambda executed successfully after being triggered manually via the aws console, expected logs could be found in kibana.
Football	Successful deploy to CODE.
Notification	Successful deploy to CODE, service successfully received a request to send a notification, no errors in logs
Notification worker lambda	Successful deploy to CODE, after sending a test notification it was observed that the harvester successfully retrieved tokens from the db and then ios and android lambdas were invoked to deliver the messages (received on real device). For these lambdas the expected logs could be found in kibana. Other services tested: registration-cleaning-worker (continued to process/delete dodgy tokens from a queue), topic-counter (continued to be triggered on expected cron, continued to retrieve/count/store topic counts), expired-registration-cleaner (successfully triggered manually via the aws console)
Registration	Successful deploy to CODE, request to register a new device was successful, corresponding token (with expected topic) could be retrieved from the database.
Report	Successful deploy to CODE, api responded successfully to /healthcheck, /notifications and /notifications/:id requests.
Report extractor	Successful deploy to CODE, lambda executed successfully after triggering it manually via the aws console, expected logs could be found in kibana (no errors).
Schedule	Successful deploy to CODE, lambda continued to be triggered every minute, logs appeared normal (no errors).
SLO monitor	Successful deploy to CODE, after sending a breaking news notification in CODE I could then see the lambda invoked successfully, along with expected logs (no unexpected errors reported).

Mark-McCracken

Excellent work! (assuming your builds all run successful, which is a big assumption).
All looks fine to me, I've left a few tiny optional comments, but you've definitely got this right.
Lovely code, I could read that all day

.github/workflows/event-consumer.yml

Mark-McCracken · 2023-08-23T13:40:34Z

.github/workflows/event-consumer.yml

+      - name: Compile, test and assembly
+        run: sbt "project eventconsumer" "compile" "test" "assembly"
+
+      - name: Upload to riff-raff
+        uses: guardian/actions-riff-raff@v2
+        with:
+          projectName: mobile-n10n:eventconsumer
+          configPath: eventconsumer/riff-raff.yaml
+          contentDirectories: |
+            eventconsumer:
+              - eventconsumer/target/scala-2.13/eventconsumer.jar


Sometimes I've had differences between what ran locally and what came out in github actions, for the version of scala, ie. 2.12 locally vs. 2.13 in GHA, and those could change in theory.
Sometimes to fix this, I've added something like this:

- name: copy jar to root dir run: cp eventconsumer/target/scala-*/eventconsumer.jar .

then in the riffraff action, provided the path at the root directory

Great idea - I've added this extra step :)

.github/workflows/fake-breaking-news.yml

.github/workflows/football.yml

Mark-McCracken · 2023-08-23T13:44:42Z

.github/workflows/registration.yml

+          cd cdk
+          npm install
+          npm run lint
+          npm test


Since you're writing tests, might be nice to add a test reporter step after this.
This is the one I've used:
https://github.com/dorny/test-reporter/tree/main

and here's an example:
https://github.com/guardian/janus/blob/e41b96a9221c0f2cf5d558990d81b3d8dd981bbc/.github/workflows/build.yml#L55-L64

I've been looking at the test reports - for the scala tests we use specs2, which output test data in the .stats format. I don't think this is supported by the action. I tested anyway and got an expected error.

For the cdk tests, I can't see the output of the tests on my local machine, so not entirely sure what path to specify for the reporter. Will do a little more digging.

.github/workflows/notification-worker-lambdas.yml

tkgnm

Lovely stuff! Thanks for doing this - this looks ginormous. Do we have a ticket to convert the non-cdk stuff in this project to cdk?

DavidLawes · 2023-08-24T10:32:54Z

Lovely stuff! Thanks for doing this - this looks ginormous. Do we have a ticket to convert the non-cdk stuff in this project to cdk?

Yep, it's a bit big, sorry! About the cdk migration there's a ticket in the backlog https://theguardian.atlassian.net/browse/LIVE-4170

DavidLawes · 2023-08-24T10:44:47Z

After deploying this change:

All checks passed

Continuous deployments to prod were successful

DavidLawes · 2023-08-24T13:46:43Z

NB: these workflows make use of some repository secrets, I've made copies of the values of these in SSM:

AWS_ROLE_ARN: param name /mobile-n10n/github-build-action/aws-role-arn
NOTIFICATION_LAMBDA_REPOSITORY_ID: param name /mobile-n10n/github-build-action/repository-id
NOTIFICATION_LAMBDA_REPOSITORY_URL: param name /mobile-n10n/github-build-action/repository-url

Also switches from deprecated sbt-riffraff-artifact plugin to devx recommendation guardian/actions-riff-raff. Based on guardian/mobile-n10n#1151 and guardian/s3-upload#53

Migrate from Teamcity to Github actions Also switches from deprecated sbt-riffraff-artifact plugin to devx recommendation guardian/actions-riff-raff. Based on guardian/mobile-n10n#1151 and guardian/s3-upload#53

DavidLawes added 30 commits August 21, 2023 09:46

create build workflow

dfc7792

change node cache and path

7e9967c

def env vars for docker publish step

5434600

login to ecr

26362ae

use deploy.sh to connect and publish

200973d

check image tags

5f150d2

print out docker images

a5aa9c6

remove inline comment

86d3ace

try explicitly updating tag

c27ed78

use riff raff role before upload

35234a2

use actions-riff-raff for event consumer

6e1dd52

create eventconsumer artifact

1e318be

split workflows

dfae3e0

update wildcard syntax

a4f9c54

define package-lock path

659b831

add missing colons

d16e690

define workflows for all services

213c7b5

Merge branch 'main' into LIVE-5304-migrate-build-to-github-actions

e7b42b9

remove pull request triggers

06d7cd2

update schedule lambda assembly command

378555e

create template json for slo monitor

beb8a70

update jar name for schedule lambda

48d6fe5

generate debian package

6a9b114

update trigger paths for notification app

0854f44

update all debian packages

5c8d4a8

make compile test and assembly step project specific

7896179

generate cdk for registration

236a77d

remove sbt riffraff artifact

fba03c2

update trigger paths

9134a0d

update worker lambda workflow

4dd6d1d

DavidLawes added 3 commits August 23, 2023 12:01

update template path for notification cfn

175f37f

update template path for report

fe8d90d

update project name for schedule lambda

b3baa0d

Mark-McCracken approved these changes Aug 23, 2023

View reviewed changes

DavidLawes added 14 commits August 23, 2023 15:08

remove superfluous branches wildcard

8b9319b

split worker lambda cdk synth into 2 steps

a5c37da

remove redundant script

ef7a981

add step to move event consumer jar to root

8565702

move jar to root for every assembly workflow

9e5d547

test reporter with specs2 tests

dc23182

update test reports path

8488d75

remove test reporter for scala tests

98c950f

define build number offset

c1dda93

update publish script for notification worker

f3cc396

don't use set-env

8a12b2f

remove rogue $

eec7a68

update syntax

b4c63d0

remove echo

43e4e00

DavidLawes marked this pull request as ready for review August 24, 2023 09:53

DavidLawes requested a review from a team August 24, 2023 09:53

tkgnm approved these changes Aug 24, 2023

View reviewed changes

DavidLawes merged commit d85cc38 into main Aug 24, 2023
2 checks passed

DavidLawes deleted the LIVE-5304-migrate-build-to-github-actions branch August 24, 2023 10:33

davidfurey mentioned this pull request Oct 5, 2023

Adopt GitHub Actions for CI guardian/story-packages#189

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate build to GitHub actions #1151

Migrate build to GitHub actions #1151

DavidLawes commented Aug 21, 2023 •

edited

Loading

Mark-McCracken left a comment

Mark-McCracken Aug 23, 2023

DavidLawes Aug 24, 2023

Mark-McCracken Aug 23, 2023

DavidLawes Aug 24, 2023

tkgnm left a comment

DavidLawes commented Aug 24, 2023

DavidLawes commented Aug 24, 2023

DavidLawes commented Aug 24, 2023

Migrate build to GitHub actions #1151

Migrate build to GitHub actions #1151

Conversation

DavidLawes commented Aug 21, 2023 • edited Loading

What does this change?

Docker lambdas

Testing

Mark-McCracken left a comment

Choose a reason for hiding this comment

Mark-McCracken Aug 23, 2023

Choose a reason for hiding this comment

DavidLawes Aug 24, 2023

Choose a reason for hiding this comment

Mark-McCracken Aug 23, 2023

Choose a reason for hiding this comment

DavidLawes Aug 24, 2023

Choose a reason for hiding this comment

tkgnm left a comment

Choose a reason for hiding this comment

DavidLawes commented Aug 24, 2023

DavidLawes commented Aug 24, 2023

DavidLawes commented Aug 24, 2023

DavidLawes commented Aug 21, 2023 •

edited

Loading