Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Integrate automated smoke test into release workflow #153

Merged
merged 3 commits into from
May 19, 2022

Conversation

adebayor123
Copy link
Member

@adebayor123 adebayor123 commented May 11, 2022

Details

Currently as part of the release process, contributors in AWS RUM manually perform smoke tests to gain assurance on the newly-releasing web client version's behavior. This process has a couple of issues:

  1. There are no predefined set of tests that contributors go through.
  2. It is easy to miss out on regressions.

To mitigate this, this PR integrates automated smoke test as part of the release workflow.

Changes

  1. Embed smoke.html into repository, with all sensitive parameters anonymized and stored in GitHub secrets.
  2. During the release workflow, the automated smoke test will run after a new version has been pushed to the CDN.
  3. The smoke test will perform a set of defined test cases on the new version of the web client.
  4. If the smoke test cases are successful, proceeds to NPM release. Otherwise, stops the release workflow.

Testing

  1. Successful Run: https://github.com/adebayor123/aws-rum-web/runs/6381491772?check_suite_focus=true
  2. Unsuccessful Run: https://github.com/adebayor123/aws-rum-web/runs/6381538124?check_suite_focus=true
  3. Final verification Run: https://github.com/adebayor123/aws-rum-web/runs/6381730545?check_suite_focus=true

Setup

This is only applicable to AWS RUM members.

Notes

When the test is run with 4 workers, all the ingested events were retrieved within a maximum of 2 minutes. However, when the tests are run with 6 workers, around 1-2 cases time out (5m) and the overall time to retrieve events is increased to 2-3 minutes.
AWS RUM team should investigate into this issue.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

playwright.config.ts Outdated Show resolved Hide resolved
@@ -0,0 +1,2 @@
bucket=$1
aws s3api put-object --bucket $bucket --key "smoke.html" --body processed_smoke.html --content-type "text/html"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Would it be difficult to version the smoke test file? E.g., smoke-1.5.1.html? This way we could (1) retain older versions for testing, and (2) it would be easy to verify which versions have been tested.

This isn't a blocker, but something to consider if it's easy to do.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to do so, I think the changes need to happen on (1) CDK to edit the bucket with versioning characteristics (2) upload_smoke_test.sh and cd.yaml.

I think it's better to have the fix incorporated with this, since otherwise we will have to keep the enhancement in our backlog (which might get overlooked).

I will add the changes with the next revision.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification: Bucket versioning is not needed - we expect the bucket to hold different versions of the smoke test, with the naming convention of *.cloudfront.net/smoke-{version}.html
Verified testing in gamma

Copy link
Member

@qhanam qhanam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm pending @limhjgrace question

@adebayor123
Copy link
Member Author

adebayor123 commented May 18, 2022

New Tests:

Verified:

  • Generates test URL based on the package version
  • Smoke test applications names include the version

@qhanam qhanam merged commit f291142 into aws-observability:main May 19, 2022
@adebayor123 adebayor123 deleted the integrate-smoke-test branch May 19, 2022 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants