-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BCDA-8360 add waf sync lambda #197
Merged
Merged
Changes from 14 commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
2bf47df
BCDA-8360 First pass WAF Sync lambda
carlpartridge 5d2212b
Better testing
carlpartridge 4eef0ab
Fix linters
carlpartridge 492f5fc
Stubbing of GHA for wafsync deploy and testing
carlpartridge c80abc8
Small corrections to GHA workflows
carlpartridge 83ded31
Minor fixes to GHA
carlpartridge fd3beaf
Security updates
carlpartridge 5fd1395
Match name from platform-ops
carlpartridge f664f69
Convert how we get DATABASE_URL
carlpartridge 298c4ba
Small code review fixes
carlpartridge 2d6220a
Update readme.md
bhagatparwinder 6f5068b
Small readme correction
carlpartridge 3434da8
Correct readme
carlpartridge bb7f2e3
Fix function name
carlpartridge 10003e6
Update log level from warning to error
carlpartridge a7d1e6b
Fix lambda runtime error
carlpartridge c2123af
Attempt to verify lambda invocation
carlpartridge 45135ff
Verify output file
carlpartridge 6506692
Temp break lambda run to test integration failure mode
carlpartridge fcd6634
Test workflow failure output
carlpartridge 7938413
Test failure file output
carlpartridge ccd78ad
Fix dev deploy
carlpartridge 6bc4289
Test result output
carlpartridge 0b62393
Fix if logic
carlpartridge bf90872
Test verify script
carlpartridge ddc9461
Check output
carlpartridge f4f7caf
Test different approach
carlpartridge 6e47c19
New approach
carlpartridge File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
name: WAF Sync Lambda Dev Deploy | ||
|
||
on: | ||
workflow_call: | ||
workflow_dispatch: | ||
push: | ||
branches: | ||
- main | ||
paths: | ||
- lambda/wafsync/** | ||
- .github/workflows/waf-sync-lambda-dev-deploy.yml | ||
|
||
jobs: | ||
deploy-to-dev: | ||
permissions: | ||
contents: read | ||
id-token: write | ||
runs-on: ubuntu-latest | ||
defaults: | ||
run: | ||
working-directory: ./lambda/wafsync | ||
environment: dev | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- uses: actions/setup-go@v5 | ||
- name: Build WAF Sync Lambda zip file | ||
run: | | ||
go build -o bootstrap main.go db.go aws.go | ||
zip function.zip bootstrap | ||
- uses: aws-actions/configure-aws-credentials@v4 | ||
with: | ||
aws-region: ${{ vars.AWS_REGION }} | ||
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-dev-github-actions | ||
- name: Upload and reload | ||
run: | | ||
aws s3 cp --no-progress function.zip \ | ||
s3://bcda-dev-api-waf-sync-function/function-${{ github.sha }}.zip | ||
aws lambda update-function-code --function-name bcda-dev-api-waf-sync \ | ||
--s3-bucket bcda-dev-api-waf-sync-function --s3-key function-${{ github.sha }}.zip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
name: WAF Sync Lambda Integration Tests | ||
|
||
on: | ||
pull_request: | ||
paths: | ||
- .github/workflows/waf-sync-lambda-integration-test.yml | ||
- .github/workflows/waf-sync-lambda-dev-deploy.yml | ||
- .github/workflows/waf-sync-lambda-test-deploy.yml | ||
- .github/workflows/waf-sync-lambda-prod-deploy.yml | ||
- lambda/wafsync/** | ||
workflow_dispatch: | ||
|
||
# Ensure we have only one integration test running at a time | ||
concurrency: | ||
group: waf-sync-lambda-integration-test | ||
|
||
jobs: | ||
# Deploy first if triggered by pull_request | ||
deploy: | ||
if: ${{ github.event_name == 'pull_request' }} | ||
uses: ./.github/workflows/waf-sync-lambda-dev-deploy.yml | ||
secrets: inherit | ||
|
||
trigger: | ||
if: ${{ always() }} | ||
needs: deploy | ||
permissions: | ||
contents: read | ||
id-token: write | ||
runs-on: ubuntu-latest | ||
defaults: | ||
run: | ||
working-directory: ./lambda/wafsync | ||
outputs: | ||
start_time: ${{ steps.invoke-lambda.outputs.STARTTIME }} | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- uses: aws-actions/configure-aws-credentials@v4 | ||
with: | ||
aws-region: ${{ vars.AWS_REGION }} | ||
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-dev-github-actions | ||
- name: Send event to trigger lambda function | ||
id: invoke-lambda | ||
run: | | ||
echo "STARTTIME=`date +%s`" >> "$GITHUB_OUTPUT" | ||
aws lambda invoke --function-name bcda-dev-api-waf-sync test-result.txt | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
name: WAFSync Lambda prod deploy | ||
|
||
on: | ||
workflow_dispatch: | ||
|
||
jobs: | ||
deploy-to-prod: | ||
permissions: | ||
contents: read | ||
id-token: write | ||
runs-on: ubuntu-latest | ||
defaults: | ||
run: | ||
working-directory: ./lambda/wafsync | ||
environment: prod | ||
steps: | ||
- uses: aws-actions/configure-aws-credentials@v4 | ||
with: | ||
aws-region: ${{ vars.AWS_REGION }} | ||
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-prod-github-actions | ||
- name: Promote lambda code from test to prod | ||
run: | | ||
aws s3 cp --no-progress \ | ||
s3://bcda-test-api-waf-sync-function/function-${{ github.sha }}.zip \ | ||
s3://bcda-prod-api-waf-sync-function/function-${{ github.sha }}.zip | ||
aws lambda update-function-code --function-name bcda-prod-api-waf-sync \ | ||
--s3-bucket bcda-prod-api-waf-sync-function --s3-key function-${{ github.sha }}.zip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
name: WAFSync Lambda test deploy | ||
|
||
on: | ||
workflow_call: | ||
workflow_dispatch: | ||
|
||
jobs: | ||
deploy-to-test: | ||
permissions: | ||
contents: read | ||
id-token: write | ||
runs-on: ubuntu-latest | ||
defaults: | ||
run: | ||
working-directory: ./lambda/wafsync | ||
environment: test | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- uses: actions/setup-go@v5 | ||
- name: Build WAF Sync Lambda zip file | ||
run: | | ||
go build -o bootstrap main.go db.go aws.go | ||
zip function.zip bootstrap | ||
- uses: aws-actions/configure-aws-credentials@v4 | ||
with: | ||
aws-region: ${{ vars.AWS_REGION }} | ||
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/bcda-test-github-actions | ||
- name: Upload and reload | ||
run: | | ||
aws s3 cp --no-progress function.zip \ | ||
s3://bcda-test-api-waf-sync-function/function-${{ github.sha }}.zip | ||
aws lambda update-function-code --function-name bcda-test-api-waf-sync \ | ||
--s3-bucket bcda-test-api-waf-sync-function --s3-key function-${{ github.sha }}.zip |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,44 +1,65 @@ | ||
module github.com/CMSgov/bcda-ssas-app | ||
|
||
go 1.19 | ||
go 1.23.1 | ||
|
||
require ( | ||
github.com/aws/aws-lambda-go v1.47.0 | ||
github.com/aws/aws-sdk-go v1.55.5 | ||
github.com/go-chi/chi/v5 v5.0.7 | ||
github.com/go-chi/render v1.0.2 | ||
github.com/golang-jwt/jwt/v4 v4.5.1 | ||
github.com/google/uuid v1.3.0 | ||
github.com/jackc/pgx/v5 v5.7.1 | ||
github.com/joho/godotenv v1.5.1 | ||
github.com/lib/pq v1.10.6 | ||
github.com/lib/pq v1.10.9 | ||
github.com/newrelic/go-agent/v3 v3.18.1 | ||
github.com/pashagolub/pgxmock/v4 v4.3.0 | ||
github.com/patrickmn/go-cache v2.1.1-0.20180815053127-5633e0862627+incompatible | ||
github.com/pborman/uuid v1.2.1 | ||
github.com/sirupsen/logrus v1.9.0 | ||
github.com/stretchr/testify v1.8.1 | ||
github.com/sirupsen/logrus v1.9.3 | ||
github.com/stretchr/testify v1.9.0 | ||
golang.org/x/crypto v0.31.0 | ||
gopkg.in/macaroon.v2 v2.1.0 | ||
gorm.io/driver/postgres v1.5.4 | ||
gorm.io/gorm v1.25.5 | ||
) | ||
|
||
require ( | ||
github.com/fsnotify/fsnotify v1.5.4 // indirect | ||
github.com/hashicorp/hcl v1.0.0 // indirect | ||
github.com/magiconair/properties v1.8.6 // indirect | ||
github.com/mitchellh/mapstructure v1.5.0 // indirect | ||
github.com/pelletier/go-toml v1.9.5 // indirect | ||
github.com/pkg/errors v0.9.1 // indirect | ||
github.com/spf13/afero v1.8.2 // indirect | ||
github.com/spf13/cast v1.5.0 // indirect | ||
github.com/spf13/jwalterweatherman v1.1.0 // indirect | ||
github.com/spf13/pflag v1.0.5 // indirect | ||
github.com/spf13/viper v1.9.0 // indirect | ||
github.com/subosito/gotenv v1.3.0 // indirect | ||
gopkg.in/ini.v1 v1.66.6 // indirect | ||
gopkg.in/yaml.v2 v2.4.0 // indirect | ||
) | ||
|
||
require ( | ||
github.com/CMSgov/bcda-app v0.0.0-20250102171642-27f10380b310 | ||
github.com/ajg/form v1.5.1 // indirect | ||
github.com/davecgh/go-spew v1.1.1 // indirect | ||
github.com/golang/protobuf v1.5.3 // indirect | ||
github.com/google/uuid v1.3.0 // indirect | ||
github.com/golang/protobuf v1.5.4 // indirect | ||
github.com/jackc/pgpassfile v1.0.0 // indirect | ||
github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9 // indirect | ||
github.com/jackc/pgx/v5 v5.5.4 // indirect | ||
github.com/jackc/puddle/v2 v2.2.1 // indirect | ||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect | ||
github.com/jackc/puddle/v2 v2.2.2 // indirect | ||
github.com/jinzhu/inflection v1.0.0 // indirect | ||
github.com/jinzhu/now v1.1.5 // indirect | ||
github.com/jmespath/go-jmespath v0.4.0 // indirect | ||
github.com/pmezard/go-difflib v1.0.0 // indirect | ||
github.com/rogpeppe/go-internal v1.10.0 // indirect | ||
github.com/stretchr/objx v0.5.0 // indirect | ||
golang.org/x/net v0.23.0 // indirect | ||
github.com/stretchr/objx v0.5.2 // indirect | ||
golang.org/x/net v0.33.0 // indirect | ||
golang.org/x/sync v0.10.0 // indirect | ||
golang.org/x/sys v0.28.0 // indirect | ||
golang.org/x/text v0.21.0 // indirect | ||
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect | ||
google.golang.org/grpc v1.56.3 // indirect | ||
google.golang.org/protobuf v1.33.0 // indirect | ||
google.golang.org/protobuf v1.34.2 // indirect | ||
gopkg.in/yaml.v3 v3.0.1 // indirect | ||
) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like each time the dev lambda is invoked, an error is being returned at the moment, but the integration test says that it's passing:
Phase: invoke Status: error Error Type: Runtime.ExitError
Could we update this so that we look for a status code and a function error, since both could indicate a problem?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its interesting the
aws lambda invoke ...
returns a json response as mentioned, however the output file does not include that information. Best I could come up with is testing that we are not seeing an error in the output file.I think checking the waf for issues is best done via cloudwatch (for your specific example) as well as checking splunk logs more generally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yup, I think we could setup the alerting around cloudwatch for sure.