Automate conformance testing

[jaypipes]

The EKS team currently runs the Sonobouy conformance test suite manually before releasing the CNI plugin. We want to both automate this process and place all test definitions into the public repo.

• Create CircleCI plumbing (use eks-charts repo as example)
• Flesh out the following manual steps into a Bash script that is executed from the CircleCI YAML file

# modify security group and open lower ports
# egress on controlplane security group (1-65535)
# ingress on worker security group (1-65535)
# use c4.xlarge node as worker node

# get sonobuoy if you don't have it
go get -u -v github.com/heptio/sonobuoy

# Deploy a Sonobuoy pod to your cluster with:
sonobuoy run

#View actively running pods:
sonobuoy status 

#To inspect the logs:
sonobuoy logs

# Once sonobuoy status shows the run as completed, copy the output directory from the main Sonobuoy pod to a local directory:
sonobuoy retrieve .

# This copies a single .tar.gz snapshot from the Sonobuoy pod into your local . directory. Extract the contents into ./results with:
mkdir ./results; tar xzf *.tar.gz -C ./results
# NOTE: The two files required for submission are located in the tarball under plugins/e2e/results/{e2e.log,junit.xml}.

# To clean up Kubernetes objects created by Sonobuoy, run:
sonobuoy delete

• ensure successful Sonobuoy test run gates merges to master

Alternately, to avoid waiting for Sonobuoy logs, we could do this instead (suggested from @wongma7):

wget https://dl.k8s.io/v1.14.8/kubernetes-test.tar.gz
tar -xzf kubernetes-test.tar.gz
./kubernetes/platforms/darwin/amd64/e2e.test --kubeconfig=$KUBECONFIG --ginkgo.focus="Conformance"

[wongma7]

I don't mind sonobuoy, but if we can achieve the same thing using the e2e.test binary directly I would prefer that mostly because it's one less potential source of bugs to worry about.

For example, something like this should be possible and we'll get the test output in real-time straight to stdout rather than having to wait for sonobuoy logs

#wget https://dl.k8s.io/v1.14.8/kubernetes-test.tar.gz
#tar -xzf kubernetes-test.tar.gz
#./kubernetes/platforms/darwin/amd64/e2e.test --kubeconfig=$KUBECONFIG --ginkgo.focus="Conformance"

[jaypipes]

I don't mind sonobuoy, but if we can achieve the same thing using the e2e.test binary directly I would prefer that mostly because it's one less potential source of bugs to worry about.

For example, something like this should be possible and we'll get the test output in real-time straight to stdout rather than having to wait for sonobuoy logs

#wget https://dl.k8s.io/v1.14.8/kubernetes-test.tar.gz
#tar -xzf kubernetes-test.tar.gz
#./kubernetes/platforms/darwin/amd64/e2e.test --kubeconfig=$KUBECONFIG --ginkgo.focus="Conformance"

Fair point, Matthew. I've changed the issue title and description accordingly.

[SaranBalaji90]

Ran

wget -qO- https://dl.k8s.io/v1.14.8/kubernetes-test.tar.gz | tar -zxvf - --strip-components=4 -C /tmp  kubernetes/platforms/darwin/amd64/e2e.test

/tmp/e2e.test --ginkgo.focus="Conformance" --kubeconfig=$KUBECONFIG -alsologtostderr --ginkgo.failFast -ginkgo.flakeAttempts 2 -ginkgo.progress -ginkgo.trace -ginkgo.skip="(should support remote command execution over websockets)|(should support retrieving logs from the container over websockets)"

Had to skip the following two test because of kubernetes/kubernetes#88064

1. should support retrieving logs from the container over websockets.
2. should support remote command execution over websockets.

Updated "flakeAttempts" to 2 so that we don't see failure on flaky tests. single test failures will still be reported as failure but test suite doesn't fail.

Added "failFast" parameter so that we will know early if something fails even with retry. Usually conformance tests takes upto 2 hours to complete.

SaranBalaji90@2264ffd has the changes required. Still testing CI integration in my repo.