Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BCDA-8360 add waf sync lambda #197

Merged
merged 28 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
2bf47df
BCDA-8360 First pass WAF Sync lambda
carlpartridge Dec 26, 2024
5d2212b
Better testing
carlpartridge Dec 27, 2024
4eef0ab
Fix linters
carlpartridge Dec 27, 2024
492f5fc
Stubbing of GHA for wafsync deploy and testing
carlpartridge Dec 27, 2024
c80abc8
Small corrections to GHA workflows
carlpartridge Dec 27, 2024
83ded31
Minor fixes to GHA
carlpartridge Dec 30, 2024
fd3beaf
Security updates
carlpartridge Dec 30, 2024
5fd1395
Match name from platform-ops
carlpartridge Dec 30, 2024
f664f69
Convert how we get DATABASE_URL
carlpartridge Jan 2, 2025
298c4ba
Small code review fixes
carlpartridge Jan 6, 2025
2d6220a
Update readme.md
bhagatparwinder Jan 3, 2025
6f5068b
Small readme correction
carlpartridge Jan 6, 2025
3434da8
Correct readme
carlpartridge Jan 6, 2025
bb7f2e3
Fix function name
carlpartridge Jan 8, 2025
10003e6
Update log level from warning to error
carlpartridge Jan 9, 2025
a7d1e6b
Fix lambda runtime error
carlpartridge Jan 13, 2025
c2123af
Attempt to verify lambda invocation
carlpartridge Jan 13, 2025
45135ff
Verify output file
carlpartridge Jan 13, 2025
6506692
Temp break lambda run to test integration failure mode
carlpartridge Jan 13, 2025
fcd6634
Test workflow failure output
carlpartridge Jan 13, 2025
7938413
Test failure file output
carlpartridge Jan 13, 2025
ccd78ad
Fix dev deploy
carlpartridge Jan 13, 2025
6bc4289
Test result output
carlpartridge Jan 13, 2025
0b62393
Fix if logic
carlpartridge Jan 13, 2025
bf90872
Test verify script
carlpartridge Jan 13, 2025
ddc9461
Check output
carlpartridge Jan 13, 2025
f4f7caf
Test different approach
carlpartridge Jan 13, 2025
6e47c19
New approach
carlpartridge Jan 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/workflows/waf-sync-lambda-dev-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: WAF Sync Lambda Dev Deploy

on:
workflow_call:
workflow_dispatch:
push:
branches:
- main
paths:
- lambda/wafsync/**
- .github/workflows/waf-sync-lambda-dev-deploy.yml

jobs:
deploy-to-dev:
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./lambda/wafsync
environment: dev
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- name: Build WAF Sync Lambda zip file
run: |
go build -o bootstrap main.go db.go aws.go
zip function.zip bootstrap
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-dev-github-actions
- name: Upload and reload
run: |
aws s3 cp --no-progress function.zip \
s3://bcda-dev-api-waf-sync-function/function-${{ github.sha }}.zip
aws lambda update-function-code --function-name bcda-dev-api-waf-sync \
--s3-bucket bcda-dev-api-waf-sync-function --s3-key function-${{ github.sha }}.zip
46 changes: 46 additions & 0 deletions .github/workflows/waf-sync-lambda-integration-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: WAF Sync Lambda Integration Tests

on:
pull_request:
paths:
- .github/workflows/waf-sync-lambda-integration-test.yml
- .github/workflows/waf-sync-lambda-dev-deploy.yml
- .github/workflows/waf-sync-lambda-test-deploy.yml
- .github/workflows/waf-sync-lambda-prod-deploy.yml
- lambda/wafsync/**
workflow_dispatch:

# Ensure we have only one integration test running at a time
concurrency:
group: waf-sync-lambda-integration-test

jobs:
# Deploy first if triggered by pull_request
deploy:
if: ${{ github.event_name == 'pull_request' }}
uses: ./.github/workflows/waf-sync-lambda-dev-deploy.yml
secrets: inherit

trigger:
if: ${{ always() }}
needs: deploy
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./lambda/wafsync
outputs:
start_time: ${{ steps.invoke-lambda.outputs.STARTTIME }}
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-dev-github-actions
- name: Send event to trigger lambda function
id: invoke-lambda
run: |
echo "STARTTIME=`date +%s`" >> "$GITHUB_OUTPUT"
aws lambda invoke --function-name bcda-dev-api-waf-sync test-result.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. if an error is returned; do we need to do any verification here or do we expect a non-zero exit code?
  2. do we want to add any additional validation for other cases where it runs successfully, but the end result might be different from what we expect; ie the update doesn't return an error, but there are no IPs in the WAF?

Copy link
Collaborator Author

@carlpartridge carlpartridge Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I believe returning nonzero is enough but Im not entirely sure how to force the function to error out in order to test. I copied a lot of this code from the DPC version, maybe its worth pinging them?
  2. I feel like that would best be solved via splunk or cloudwatch alerts? They could watch for a variety of things based off the logs? I can create a ticket to look into this, either creating new alerts or modifying the jenkins alerts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It looks like it should return a status code and function error (among a few other fields), if any occurred.

It looks like each time the dev lambda is invoked, an error is being returned at the moment, but the integration test says that it's passing: Phase: invoke Status: error Error Type: Runtime.ExitError

Could we update this so that we look for a status code and a function error, since both could indicate a problem?

  1. We could add a cli command to pull a list from the waf and if it's length is zero, then we could fail?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. It looks like it should return a status code and function error (among a few other fields), if any occurred.

It looks like each time the dev lambda is invoked, an error is being returned at the moment, but the integration test says that it's passing: Phase: invoke Status: error Error Type: Runtime.ExitError

Could we update this so that we look for a status code and a function error, since both could indicate a problem?

  1. We could add a cli command to pull a list from the waf and if it's length is zero, then we could fail?

Its interesting the aws lambda invoke ... returns a json response as mentioned, however the output file does not include that information. Best I could come up with is testing that we are not seeing an error in the output file.

I think checking the waf for issues is best done via cloudwatch (for your specific example) as well as checking splunk logs more generally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, I think we could setup the alerting around cloudwatch for sure.

27 changes: 27 additions & 0 deletions .github/workflows/waf-sync-lambda-prod-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: WAFSync Lambda prod deploy

on:
workflow_dispatch:

jobs:
deploy-to-prod:
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./lambda/wafsync
environment: prod
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-prod-github-actions
- name: Promote lambda code from test to prod
run: |
aws s3 cp --no-progress \
s3://bcda-test-api-waf-sync-function/function-${{ github.sha }}.zip \
s3://bcda-prod-api-waf-sync-function/function-${{ github.sha }}.zip
aws lambda update-function-code --function-name bcda-prod-api-waf-sync \
--s3-bucket bcda-prod-api-waf-sync-function --s3-key function-${{ github.sha }}.zip
33 changes: 33 additions & 0 deletions .github/workflows/waf-sync-lambda-test-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: WAFSync Lambda test deploy

on:
workflow_call:
workflow_dispatch:

jobs:
deploy-to-test:
permissions:
contents: read
id-token: write
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./lambda/wafsync
environment: test
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- name: Build WAF Sync Lambda zip file
run: |
go build -o bootstrap main.go db.go aws.go
zip function.zip bootstrap
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-test-github-actions
- name: Upload and reload
run: |
aws s3 cp --no-progress function.zip \
s3://bcda-test-api-waf-sync-function/function-${{ github.sha }}.zip
aws lambda update-function-code --function-name bcda-test-api-waf-sync \
--s3-bucket bcda-test-api-waf-sync-function --s3-key function-${{ github.sha }}.zip
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ test:
$(MAKE) smoke-test
$(MAKE) migrations-test

setup-tests:
# Clean up any existing data to ensure we spin up container in a known state.
docker compose -f docker-compose.test.yml rm -fsv tests
docker compose -f docker-compose.test.yml build tests

# make test-path TEST_PATH="bcdaworker/worker/*.go"
test-path: setup-tests
@docker compose -f docker-compose.test.yml run --rm tests go test -v $(TEST_PATH)

load-fixtures:
docker compose -f docker-compose.migrate.yml run --rm migrate -database "postgres://postgres:toor@db:5432/bcda?sslmode=disable" -path /go/src/github.com/CMSgov/bcda-ssas-app/db/migrations up
docker compose -f docker-compose.yml run ssas sh -c 'ssas --add-fixture-data'
Expand Down
47 changes: 34 additions & 13 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,44 +1,65 @@
module github.com/CMSgov/bcda-ssas-app

go 1.19
go 1.23.1

require (
github.com/aws/aws-lambda-go v1.47.0
github.com/aws/aws-sdk-go v1.55.5
github.com/go-chi/chi/v5 v5.0.7
github.com/go-chi/render v1.0.2
github.com/golang-jwt/jwt/v4 v4.5.1
github.com/google/uuid v1.3.0
github.com/jackc/pgx/v5 v5.7.1
github.com/joho/godotenv v1.5.1
github.com/lib/pq v1.10.6
github.com/lib/pq v1.10.9
github.com/newrelic/go-agent/v3 v3.18.1
github.com/pashagolub/pgxmock/v4 v4.3.0
github.com/patrickmn/go-cache v2.1.1-0.20180815053127-5633e0862627+incompatible
github.com/pborman/uuid v1.2.1
github.com/sirupsen/logrus v1.9.0
github.com/stretchr/testify v1.8.1
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
golang.org/x/crypto v0.31.0
gopkg.in/macaroon.v2 v2.1.0
gorm.io/driver/postgres v1.5.4
gorm.io/gorm v1.25.5
)

require (
github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/magiconair/properties v1.8.6 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/spf13/afero v1.8.2 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.9.0 // indirect
github.com/subosito/gotenv v1.3.0 // indirect
gopkg.in/ini.v1 v1.66.6 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
)

require (
github.com/CMSgov/bcda-app v0.0.0-20250102171642-27f10380b310
github.com/ajg/form v1.5.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9 // indirect
github.com/jackc/pgx/v5 v5.5.4 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rogpeppe/go-internal v1.10.0 // indirect
github.com/stretchr/objx v0.5.0 // indirect
golang.org/x/net v0.23.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
golang.org/x/net v0.33.0 // indirect
golang.org/x/sync v0.10.0 // indirect
golang.org/x/sys v0.28.0 // indirect
golang.org/x/text v0.21.0 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
google.golang.org/grpc v1.56.3 // indirect
google.golang.org/protobuf v1.33.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading
Loading