Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run conformance test as part of PR/Release certification #851

Merged
merged 1 commit into from
Feb 26, 2020

Conversation

SaranBalaji90
Copy link
Contributor

@SaranBalaji90 SaranBalaji90 commented Feb 25, 2020

Description of changes:

This changes updates CircleCI config.yaml to add following functionalities:

  • Runs conformance tests as part of existing integration tests. This will now act as a gate keeper before merging the changes to master.
  • Automatically trigger integration tests when new tags are pushed. Today AmazonEKS runs tests internally when new git tag is released. With this change when new git tag is released, it will run integration tests through circleCI.
  • Adds nightly test run on master branch. With this we can catch regressions.

Addresses Issue: #686

Upcoming changes:

  • Run only tests that are required for our CNI testing when merging PR. Conformance tests takes about 2 hours to complete and some of the test might not be necessary for our CNI plugin for eg testing kubectl etc.
  • Run nightly run on branches that we have already cut. This will help us to catch regressions on already released versions.

Ran this on my personal repo - https://circleci.com/api/v1.1/project/github/SaranBalaji90/amazon-vpc-cni-k8s-1/78/output/108/0?file=true&allocation-id=5e552aea36810238d4b33f5d-0-build%2F50024F6D

Adding some of my learning here to help others in future:

  • If tags are used as filters in one of the job in any workflow, then all the jobs needs to have same tag filters.
  • CircleCI needs to be in individual branch to execute tests against those branches.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@SaranBalaji90 SaranBalaji90 changed the title circleCI configuration update to run more tests as listed below: Run conformance test as part of PR/Release certification Feb 25, 2020
Copy link
Contributor

@jaypipes jaypipes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great stuff, Balaji! Just a couple small requests inline.

Thanks!
-jay

scripts/run-integration-tests.sh Outdated Show resolved Hide resolved
@@ -50,3 +54,25 @@ workflows:
- integration_test:
requires:
- hold

# runs integration test when new git tag is pushed.
run-test-on-tags:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this and the nightly-test-run job, you will want to set:

       environment:
         RUN_CONFORMANCE: "true"

scripts/run-integration-tests.sh Show resolved Hide resolved
@SaranBalaji90 SaranBalaji90 self-assigned this Feb 25, 2020
@SaranBalaji90 SaranBalaji90 linked an issue Feb 25, 2020 that may be closed by this pull request
3 tasks
Copy link
Contributor

@jaypipes jaypipes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 :)

* Run conformance testing as part of integration test.
* Trigger integration test on new tags.
* Add nightly runs on master branch
@SaranBalaji90
Copy link
Contributor Author

SaranBalaji90 commented Feb 26, 2020

For some reason, circleCI was triggering build in my personal account instead of triggering in cni repo aws account. For now, stopped builds on my personal acct and its now building in cni repo aws account.

@jaypipes jaypipes merged commit 8cf87c0 into aws:master Feb 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automate conformance testing
2 participants