Reduce time to create EKS cluster in integration tests #786
Assignees
Labels
Comments
|
For the echo Job stuff, looks like we can configure this to run a smaller set of iterations: |
jaypipes
added a commit
to jaypipes/amazon-vpc-cni-k8s
that referenced
this issue
Jan 3, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
jaypipes
added a commit
to jaypipes/amazon-vpc-cni-k8s
that referenced
this issue
Jan 17, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
jaypipes
added a commit
to jaypipes/amazon-vpc-cni-k8s
that referenced
this issue
Jan 17, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
mogren
pushed a commit
that referenced
this issue
Jan 20, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue #686 Issue #784 Issue #786
mogren
pushed a commit
to mogren/amazon-vpc-cni-k8s
that referenced
this issue
Jan 31, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786 (cherry picked from commit 8291df3)
jaypipes
added a commit
that referenced
this issue
Jan 31, 2020
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue #686 Issue #784 Issue #786 (cherry picked from commit 8291df3)
This was referenced Sep 4, 2020
|
@sushrk is working on replacing k8stester with eksctl. |
|
#1899 replaced aws-k8s-tester with eksctl. With eksctl there is a reduction from 30-35 mins to 15-18 mins. |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Functionally testing the CNI plugin is cumbersome due to the amount of time that it takes to get the
aws-k8s-tester create clusterto complete (as well as the amount of time it takes to delete that cluster.I've noticed that the average time for the
aws-k8s-tester create clustercommand to complete seems to be around 30-35 minutes. However, a good chunk (maybe 10-15 minutes) of this time is actuallyaws-k8s-testerrunning a series of tests to "warm up" the cluster. These tests create 30Jobs spread over the 3 worker nodes. EachJobdoes a simpleechoand there is a poll collecting how manyJobs completed successfully.This warm up test is then repeated for 1000
Jobs of the same type. This takes around 8 minutes or so.I think for the purposes of the CNI integration tests, we should be able to do a couple things to speed up the iterative cycle of dev-test:
aws-k8s-testerconfiguration to run a smaller set of warm-upJobs (see 1)aws-k8s-testerto allow customized expo-backoff polling timeouts and intervals. The intervals for many polling operations is 1 minute, which meansaws-k8s-testeris basically just sitting there waiting for a minute at a time checking to see if some CloudFormation stack has had its status shifted to an expected statusaws-k8s-tester create clustercommand is even run. This means that iterative dev-test is less cumbersome for tracking down failures in the script itself.The text was updated successfully, but these errors were encountered: