Automation script for running IT #1759

Shreya027 · 2021-11-19T21:42:57Z

What type of PR is this?
feature

Which issue does this PR fix:
Automates running of integration tests in the integration-new folder.

What does this PR do / Why do we need it:

Iterates and runs tests in new integration folder having sub-folders
Takes an optional flag -f in order to provide FOCUS as a parameter to run selective tests
Uses a set of ENV variables and derives other variables needed to run tests
Created a new folder test inside script, all refactored test scripts will be added here in future.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Testing done on this change:

Running script with focus flag as "CANARY"


% ./scripts/test/run-integration-tests.sh -f "CANARY"
/Volumes/workplace/go/src/github.com/amazon-vpc-cni-k8s/test/integration-new /Volumes/workplace/go/src/github.com/amazon-vpc-cni-k8s
Running integration test with the following configuration:
KUBE CONFIG: /Users/shrenaik/.kube/eksctl/clusters/test-cluster-ec2
CLUSTER_NAME: test-cluster-ec2
AWS_REGION: us-west-2
OS_OVERRIDE: darwin
FOCUS: CANARY
Running Suite: CNI Pod Networking Suite
=======================================
Random Seed: 1637357115
Will run 6 of 15 specs

STEP: creating test namespace
STEP: getting the node with the node label key eks.amazonaws.com/nodegroup and value

Running script with no focus flag passed


% ./scripts/test/run-integration-tests.sh            
/Volumes/workplace/go/src/github.com/amazon-vpc-cni-k8s/test/integration-new /Volumes/workplace/go/src/github.com/amazon-vpc-cni-k8s
Running integration test with the following configuration:
KUBE CONFIG: /Users/shrenaik/.kube/eksctl/clusters/test-cluster-ec2
CLUSTER_NAME: test-cluster-ec2
AWS_REGION: us-west-2
OS_OVERRIDE: darwin
Running Suite: CNI Pod Networking Suite
=======================================
Random Seed: 1637357132
Will run 15 of 15 specs

STEP: creating test namespace
STEP: getting the node with the node label key eks.amazonaws.com/nodegroup and value

Automation added to e2e:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:

Does this change require updates to the CNI daemonset config files to work?:

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

jayanthvn · 2021-11-30T00:12:14Z

scripts/test/run-integration-tests.sh

+
+set -e
+OS_OVERRIDE=$(go env GOOS)
+pushd ./test/integration-new


Once we have all the PRs merged, lets replace integration-new -> integration

jayanthvn

Lets add it in the existing nightly workflow - https://github.com/aws/amazon-vpc-cni-k8s/blob/master/.github/workflows/cron-test.yml along with the existing integration test run.

Once we have successful runs then we can add it here - https://github.com/aws/amazon-vpc-cni-k8s/blob/master/.github/workflows/integration-tests.yml which will be tested for each PR.

abhipth · 2021-11-30T01:15:21Z

scripts/test/run-integration-tests.sh

+KUBE CONFIG: $KUBECONFIG
+CLUSTER_NAME: $CLUSTER_NAME
+AWS_REGION: $AWS_REGION
+OS_OVERRIDE: $OS_OVERRIDE"


We can remove this variable, as it's not required.

scripts/test/run-integration-tests.sh

abhipth · 2021-11-30T01:25:35Z

scripts/test/run-integration-tests.sh

+
+    cd "${dir%*/}"  
+    if [ -n "$FOCUS" ]; then
+        ginkgo --focus="$FOCUS" -v -r -- --cluster-kubeconfig=$KUBECONFIG --cluster-name=$CLUSTER_NAME --aws-region=$AWS_REGION --aws-vpc-id=$VPC_ID


with set -e if the firs test fails then all the other tests will be skipped as the script will exit immediately. It would be nice if we can allow all suites to run and later return with non zero exit code if any one suite failed.

Did something similar here - https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/master/scripts/test/test-with-eksctl.sh#L136-L142

abhipth · 2021-11-30T19:30:32Z

.github/workflows/cron-test.yml

+
+      - name: Run new integration tests
+        run: |
+          ./scripts/test/run-integration-tests.sh 


Will the script work without any environment variables?
Also, I prefer doing this as part of a different PR so it's easy to revert if there are issues.

@Shreya027 - You can test this on your local fork. Lets add the new script/test cases for nightly runs since it has been long pending.

This VlanId will appear in the prevResult during cmdDel request CleanUp Pod Network using vlanId from prevResult in CNI itself No need to call ipamd Log formatting changes Added hostNetworking Setup test for pods using security groups Updated cleanUpPodENI method Skip processing Delete request if prevResult is nil Add Logging vlanId to ipamd Add support to test with containerd nodegroup in pod-eni test Add check for empty Netns() in cni Manifests and Readme updates (aws#1732) * Manifests and Readme updates * update manifest.jsonnet Readme updates (aws#1735) Updates to troubleshooting doc (aws#1737) * Updates to troubleshooting doc * updates to troubleshooting doc imdsv2 changes (aws#1743) fix flaky canary test (aws#1742) add CODEOWNERS (aws#1747) Snat tests: [agent is already updated] (aws#1513) * resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Minor change to container runtime argument Check for Empty NetNs() first Fallback to older method if prevResult is nil Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file

Add VlanId in the cmdAdd Result struct This VlanId will appear in the prevResult during cmdDel request CleanUp Pod Network using vlanId from prevResult in CNI itself No need to call ipamd Log formatting changes Added hostNetworking Setup test for pods using security groups Updated cleanUpPodENI method Skip processing Delete request if prevResult is nil Add Logging vlanId to ipamd Add support to test with containerd nodegroup in pod-eni test Add check for empty Netns() in cni Manifests and Readme updates (aws#1732) * Manifests and Readme updates * update manifest.jsonnet Readme updates (aws#1735) Updates to troubleshooting doc (aws#1737) * Updates to troubleshooting doc * updates to troubleshooting doc imdsv2 changes (aws#1743) fix flaky canary test (aws#1742) add CODEOWNERS (aws#1747) Snat tests: [agent is already updated] (aws#1513) * resolved conflicts with go.sum * Updated test agent image * Removed redundant files * Addressed PR comments Fixed go.sum in root folder Changed DescribeInstanceWithFilter to DescribeInstances Moved GetPrimaryInstanceId from ec2 interface Added GinkgoWriter Updated Readme for Snat test Rearranged snat_test logic Updated Readme for test/e2e * Minor change to logging Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Minor change to container runtime argument Check for Empty NetNs() first Fallback to older method if prevResult is nil Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file

Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file

Fix compilation errors (aws#1751) add support for running canary script in different regions (aws#1752) Regenerate pod eni values for new instance types (aws#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Closed issue message (aws#1761) * closed issue message * update message fix typo in upload script (aws#1763) Update calico file path Use an unique s3 bucket name (aws#1760) Update region Workflow to build arm and x86 images (aws#1764) DataStore.GetStats() refactoring to simplify adding new fields (aws#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (aws#1770) Automation script for running IT (aws#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (aws#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (aws#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (aws#1777) Chart and Manifest updates (aws#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (aws#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file Add HostNetworking Test for PPSG in test agent (aws#1720) * Add HostNetworking Test for PPSG in test agent * Updated PPSG test to validate vlan.eth link

…1715) * merge conflicts with cni-metrics-helper chart Fix compilation errors (#1751) add support for running canary script in different regions (#1752) Regenerate pod eni values for new instance types (#1754) * Regenerate pod eni values for new instance types Co-authored-by: Senthil Kumaran <senthilx@amazon.com> Closed issue message (#1761) * closed issue message * update message fix typo in upload script (#1763) Update calico file path Use an unique s3 bucket name (#1760) Update region Workflow to build arm and x86 images (#1764) DataStore.GetStats() refactoring to simplify adding new fields (#1704) * DataStore.GetStats() refactoring to simplify adding new fields * cleanup * cleanup * cleanup * goimports * rename test to TestGetStatsV4 * address comments * fix typo * update * update "IP pool is too low" logging * GetStats() -> GetIpStats() * GetStats() -> GetIpStats() in tests and comments * update test * cleanup test * add logPoolStats comment Fix KOPS_STATE_STORE (#1770) Automation script for running IT (#1759) Update issue template Update issue template with email address Update issue template Update go.mod for integration folder (#1741) * Update go.mod for integration folder - Update go.mod for integration folder * Change integration test to use new K8s test framework * Modify server pod image * Switch to Nginx port 80 for server pod * Switch server port in client test * Remove custom command directive for Nginx pod * Added ping command for host checks README: mention arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy (#1768) Co-authored-by: Shreya027 <shrenaik@amazon.com> Add dl1.24xlarge to ENILimits override list (#1777) Chart and Manifest updates (#1771) * Chart and Manifest updates * Update probe timeout values Change workflow to use git install (#1785) - Change workflow to use git install as the go get command was altering go.mod file without updating go.sum file Add HostNetworking Test for PPSG in test agent (#1720) * Add HostNetworking Test for PPSG in test agent * Updated PPSG test to validate vlan.eth link * Minor change to logging CLUSTER_ID and Region values Fix for cni-metrics-helper failing integration test * Fixed merge conflicts * Readme update * Updated Readme with more description of AWS_CLUSTER_ID * minor change

Shreya Anil Naik added 2 commits November 19, 2021 13:27

Add automation for running IT

4920d0f

Add recursive flag -r to ginkgo command

22febaf

Shreya027 requested a review from a team as a code owner November 19, 2021 21:42

Shreya027 requested review from jayanthvn and abhipth November 19, 2021 21:44

jayanthvn reviewed Nov 30, 2021

View reviewed changes

abhipth suggested changes Nov 30, 2021

View reviewed changes

jayanthvn mentioned this pull request Nov 30, 2021

Count addresses in cooldown against used ones in datastoreTargetState() #1700

Closed

Shreya027 added 2 commits November 30, 2021 23:38

Merge branch 'aws:master' into cni-it

a25cd2e

Add ENDPOINT env var, TEST_FAILED status and other minor refactoring

f87037a

abhipth suggested changes Nov 30, 2021

View reviewed changes

Revert cron-test file changes

28d11eb

Shreya027 force-pushed the cni-it branch from bc6fb70 to 28d11eb Compare November 30, 2021 19:54

abhipth approved these changes Nov 30, 2021

View reviewed changes

abhipth merged commit 949740d into aws:master Nov 30, 2021

haouc pushed a commit to haouc/amazon-vpc-cni-k8s that referenced this pull request Feb 9, 2022

Automation script for running IT (aws#1759)

5809534

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automation script for running IT #1759

Automation script for running IT #1759

Shreya027 commented Nov 19, 2021 •

edited

Loading

jayanthvn Nov 30, 2021

jayanthvn left a comment

abhipth Nov 30, 2021

abhipth Nov 30, 2021

abhipth Nov 30, 2021

jayanthvn Dec 1, 2021

Automation script for running IT #1759

Automation script for running IT #1759

Conversation

Shreya027 commented Nov 19, 2021 • edited Loading

jayanthvn Nov 30, 2021

Choose a reason for hiding this comment

jayanthvn left a comment

Choose a reason for hiding this comment

abhipth Nov 30, 2021

Choose a reason for hiding this comment

abhipth Nov 30, 2021

Choose a reason for hiding this comment

abhipth Nov 30, 2021

Choose a reason for hiding this comment

jayanthvn Dec 1, 2021

Choose a reason for hiding this comment

Shreya027 commented Nov 19, 2021 •

edited

Loading