Add Kubeadm-DinD-Cluster based kubetest deployer for IPv6 CI #7529

leblancd · 2018-04-03T17:50:14Z

This change adds a Kubeadm-DinD-Cluster based kubetest deployer that will be used to create a
CI for Kubernetes IPv6-only functionality.

The Prow job config for this CI will be added via a separate PR (see PR #9640).

This CI test uses deployment scripts from the kubernetes-sigs/kubeadm-dind-cluster
github (K-D-C) repo to set up a containerized (Docker-in-Docker-in-Docker), multi-node Kubernetes
cluster that can be run Prow container.

The K-D-C deployment scripts provide some built-in functionality that is important for testing IPv6-only support (see https://github.com/leblancd/kube-v6 for details):

Bridge CNI and host local IPAM plugins are loaded (both plugins support IPv6)
Static routes are provided from each node to pod networks on other nodes.
When the scripts are run in IPv6 mode, the following is configured for the cluster:
- IPv6 node addresses and service subnets
- IPv6 CNI configuration for pods
- IPv6 DNS nameserver in kubelet startup config
- Bind9 container for DNS64 functionality
- Tayga container for NAT64 functionality

The End-to-End test suite that is used to test functionality is described here:
https://github.com/CiscoSystems/kube-v6-test

I've tested this CI by running its Prow container locally (via 'docker run ...'), sample results and logs can be found here

Note: With this current change set, kubeadm-dind-cluster always does its own build of Kubernetes.
As a potential future CI performance improvement, this can be changed to have kubeadm-dind-cluster use a shared build from GCS.

krzyzacy · 2018-04-03T17:51:24Z

/assign @Q-Lee

fejta · 2018-04-03T17:51:32Z

/assign @Q-Lee

BenTheElder · 2018-04-03T18:17:19Z

prow/config.yaml

+          secret:
+            secretName: ssh-key-secret
+            defaultMode: 0400
+        - name: cache-ssd


do not use this.

BenTheElder · 2018-04-03T18:17:38Z

prow/config.yaml

+            mountPath: /etc/ssh-key-secret
+            readOnly: true
+          - name: cache-ssd
+            mountPath: /root/.cache


please remove this mount and the containerPort, I've just finished eliminating this.

Will remove these, thanks.

BenTheElder · 2018-04-03T18:23:10Z

that is running on a GCE instance.

I don't see anything around selecting a project or existing VM so I assume this brings up a VM in the current project. I don't think this works well with our resource cleanup (IE boskos etc) @krzyzacy

We need to have something that is guaranteed to be cleaned up later even if the job aborts.

leblancd · 2018-04-03T18:34:48Z

@BenTheElder - Yes, it creates a GCE VM in the current project via docker-machine create. I'm relying on Boskos to pick the project (gcp-project isn't defined in the jobs config). Do I need to register some cleanup with Boskos?

krzyzacy · 2018-04-03T18:46:32Z

@leblancd if it's just a gce instance then boskos can handle it fine.

you can run locally with ./boskos/janitor/janitor.py --project=your-sandbox --hours=0 to verify if janitor is able to clean up your project.

krzyzacy · 2018-04-03T19:50:30Z

/ok-to-test

Q-Lee · 2018-04-03T22:33:38Z

images/gce-dind/Dockerfile

+    apt-get clean
+
+# Install kubeadm-dind-cluster scripts
+RUN cd /root && \


Rather than baking it in, we should pull git repos at point of use, so it's easier to change versions.

we can do this with a second --repo bootstrap flag on the job instead of baking it into an image

Good idea, thanks. Will change this.

Done. Added a --repo flag for the kubeadm-dind-cluster repo.

Q-Lee · 2018-04-03T22:34:49Z

images/gce-dind/Dockerfile

+    git clone -b k8s_ipv6_ci https://github.com/leblancd/kubeadm-dind-cluster.git
+
+# Install docker-machine
+RUN curl -L https://github.com/docker/machine/releases/download/v0.13.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && install /tmp/docker-machine /usr/local/bin/docker-machine


Why do we need docker machine, and shouldn't this be managed by apt-get?

The kubeadm-dind-cluster scripts use docker-machine to both create the instance on GCE, and to create a connection to the docker daemon running on that instance. Once the docker-machine connection is set up, the docker client in the Prow container will use the docker daemon in the GCE instance, e.g. for commands like 'docker run ...', 'docker ps ...', etc.

The recommended way to install docker-machine is to curl the binary.

Q-Lee · 2018-04-03T22:53:08Z

kubetest/gce_dind.go

+// to create a containerized, Docker-in-Docker (DinD) based Kubernetes cluster
+// that is running in a remote GCE instance.
+
+package main


We're trying to avoid putting new kubetest implementations into the main package. It's been a non-trivial effort to begin teasing them apart.

Will move this out of the main package.

Done. Moved gce_dind.go to its own "gcedind" package (under kubetest).

leblancd · 2018-10-15T16:17:32Z

@amwat , thanks for the review! I moved the build of test binaries to TestSetup() as suggested. I also made a small change to eliminate the use of the DIND_STORAGE_DRIVER environmental variable when running the K-D-C script, and therefore was able to get rid of the deployer.runWithEnv() method.
Tested locally: results.

This change adds prow config for a CI test job for testing Kubernetes IPv6-only functionality. This config uses the Kubeadm-DinD-Cluster (K-D-C) kubetest deployer that was added via PR kubernetes#7529. The K-D-C scripts provide some built-in functionality that is important for testing IPv6-only support: Bridge CNI and host local IPAM plugins are loaded (both plugins support IPv6) Static routes are provided from each node to pod networks on other nodes. When the scripts are run in IPv6 mode, the following is configured for the cluster: IPv6 node addresses and service subnets IPv6 CNI configuration for pods IPv6 DNS nameserver in kubelet startup config Bind9 container for DNS64 functionality Tayga container for NAT64 functionality The End-to-End test suite that is used to test functionality is described here: https://github.com/CiscoSystems/kube-v6-test I've tested this CI by running its Prow container locally (via 'docker run ...'), sample results and logs can be found here: https://k8s-gubernator.appspot.com/builds/my-kubernetes-jenkins/pull/65951/pull-kubernetes-e2e-kubeadm-dind-ipv6/

krzyzacy · 2018-10-15T17:51:24Z

kubetest/kubeadmdind/BUILD.bazel

@@ -0,0 +1,30 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")


also add an OWNERS file for kubeadmdind

@krzyzacy - Whom should I list as reviewers/approvers, and do we need labels in the OWNERS file?

no labels are needed, and I would put yourself & people from your project as reviewers/approvers (don't need to put test-infra people here since we are already top-level approvers)

Will add, thanks.

krzyzacy · 2018-10-15T17:55:40Z

I think we should unblock this? (I'm fine since the deployer code are within it's own package)... maybe also worth make an issue to make ipv6 work with kind if we have future plans? @BenTheElder ^^

krzyzacy · 2018-10-16T18:24:58Z

kubetest/main.go

+	if o.ipMode == "ipv6" || o.ipMode == "dual-stack" {
+		log.Printf("Enabling IPv6")
+		if err := control.FinishRunning(exec.Command("sysctl", "-w", "net.ipv6.conf.all.disable_ipv6=0")); err != nil {
+			return err


last nit: seems (at least now) only kubeadmdind deployer cares about the ipMode flag? Do you think it makes more sense to put it into kubeadmdind package?

@krzyzacy - Will we need ipMode for KinD deployment?

I don't know the design about that yet, I would hope each deployer can handle specific logic like this rather than in main.go...

Okay, I'll move this flag to kubeadmdind.

krzyzacy · 2018-10-16T18:25:50Z

/lint

k8s-ci-robot

@krzyzacy: 0 warnings.

In response to this:

/lint

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This change adds a CI test job for testing Kubernetes IPv6-only functionality. This CI test uses deployment scripts from the kubernetes-sigs/kubeadm-dind-cluster (K-D-C) github repo to set up a multi-node, containerized (Docker-in-Docker-in-Docker, or DinDinD) Kubernetes cluster that is running inside a Prow container. (A containerized environment is needed because the Prow infrastructure does not support IPv6 connectivity.) Note that when the K-D-C scripts are used to spin up a Kubernetes cluster in IPv6 mode, it will create the following container on the GCE instance: - bind9 container that performs DNS64 functionality - tayga container that performs NAT64 functionality - kube-master node container - kube-node-1 node container - kube-node-2 node container K-D-C uses the IPv6-capable Bridge CNI network plugin for inter-node connectivity. The K-D-C scripts also add host routes on each worker node container so that pod networks on worker pod networks are reachable, and configure IPv6 pod, service, and node addresses/prefixes.

leblancd · 2018-10-17T19:20:29Z

@krzyzacy - I moved the IP mode flag to make it kubeadm-dind specific, added a UT test case for IP mode validation, and made a minor improvement to the existing UT (replaced if/else with case statement).
(Latest local test run)

krzyzacy · 2018-10-17T19:30:42Z

great! thanks!
/lint

k8s-ci-robot

@krzyzacy: 0 warnings.

In response to this:

great! thanks!
/lint

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

krzyzacy · 2018-10-17T19:31:06Z

/lgtm

k8s-ci-robot · 2018-10-17T19:31:07Z

LGTM label has been added.

Git tree hash: 80bb9b8f0f1378914a28a79b657854d14eaa33f1

k8s-ci-robot · 2018-10-17T19:31:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: krzyzacy, leblancd, MrHohn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [krzyzacy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

krzyzacy · 2018-10-17T19:32:23Z

@leblancd feel free to prep a PR for prowjob config, preferably a periodic job, I'll push a new kubetest image later today.

This change adds prow config for a CI test job for testing Kubernetes IPv6-only functionality. This config uses the Kubeadm-DinD-Cluster (K-D-C) kubetest deployer that was added via PR kubernetes#7529. The K-D-C scripts provide some built-in functionality that is important for testing IPv6-only support: Bridge CNI and host local IPAM plugins are loaded (both plugins support IPv6) Static routes are provided from each node to pod networks on other nodes. When the scripts are run in IPv6 mode, the following is configured for the cluster: IPv6 node addresses and service subnets IPv6 CNI configuration for pods IPv6 DNS nameserver in kubelet startup config Bind9 container for DNS64 functionality Tayga container for NAT64 functionality The End-to-End test suite that is used to test functionality is described here: https://github.com/CiscoSystems/kube-v6-test I've tested this CI by running its Prow container locally (via 'docker run ...'), sample results and logs can be found here: https://k8s-gubernator.appspot.com/builds/my-kubernetes-jenkins/pull/65951/pull-kubernetes-e2e-kubeadm-dind-ipv6/

leblancd · 2018-10-17T20:13:51Z

@krzyzacy - I have a PR for a presubmit prowjob config. BenTheElder mentioned in this comment on Aug 27 that there was a way to manually trigger presubmit jobs with a fake PR (and job is configured with always-run=false, so it won't auto trigger). If manually triggering a presubmit job isn't practical, let me know, and I'll convert the prow config to a periodic job.

This change adds prow config for a CI test job for testing Kubernetes IPv6-only functionality. This config uses the Kubeadm-DinD-Cluster (K-D-C) kubetest deployer that was added via PR kubernetes#7529. The K-D-C scripts provide some built-in functionality that is important for testing IPv6-only support: Bridge CNI and host local IPAM plugins are loaded (both plugins support IPv6) Static routes are provided from each node to pod networks on other nodes. When the scripts are run in IPv6 mode, the following is configured for the cluster: IPv6 node addresses and service subnets IPv6 CNI configuration for pods IPv6 DNS nameserver in kubelet startup config Bind9 container for DNS64 functionality Tayga container for NAT64 functionality The End-to-End test suite that is used to test functionality is described here: https://github.com/CiscoSystems/kube-v6-test I've tested this CI by running its Prow container locally (via 'docker run ...'), sample results and logs can be found here: https://k8s-gubernator.appspot.com/builds/my-kubernetes-jenkins/pull/65951/pull-kubernetes-e2e-kubeadm-dind-ipv6/

leblancd requested review from BenTheElder and krzyzacy as code owners April 3, 2018 17:50

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 3, 2018

k8s-ci-robot assigned Q-Lee Apr 3, 2018

leblancd force-pushed the kubeadm_dind_prow branch from fe37505 to 2714be9 Compare April 3, 2018 17:52

BenTheElder reviewed Apr 3, 2018

View reviewed changes

prow/config.yaml Outdated

secret:

secretName: ssh-key-secret

defaultMode: 0400

- name: cache-ssd

Copy link

Member

BenTheElder Apr 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not use this.

BenTheElder reviewed Apr 3, 2018

View reviewed changes

leblancd force-pushed the kubeadm_dind_prow branch from 2714be9 to 18f4001 Compare April 3, 2018 18:51

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 3, 2018

leblancd force-pushed the kubeadm_dind_prow branch from 18f4001 to 1edadfc Compare April 3, 2018 21:43

Q-Lee reviewed Apr 3, 2018

View reviewed changes

leblancd force-pushed the kubeadm_dind_prow branch 2 times, most recently from c06ae3e to ad9952f Compare April 4, 2018 19:03

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 4, 2018

leblancd force-pushed the kubeadm_dind_prow branch from ad9952f to 23bd255 Compare April 4, 2018 20:31

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 4, 2018

leblancd force-pushed the kubeadm_dind_prow branch 4 times, most recently from 61d5e0e to d20cdb9 Compare April 8, 2018 23:00

leblancd force-pushed the kubeadm_dind_prow branch from 2437afe to b45c3b8 Compare October 15, 2018 16:03

krzyzacy reviewed Oct 15, 2018

View reviewed changes

leblancd force-pushed the kubeadm_dind_prow branch 2 times, most recently from 37a54c4 to c03d2fc Compare October 15, 2018 20:06

krzyzacy reviewed Oct 16, 2018

View reviewed changes

k8s-ci-robot reviewed Oct 16, 2018

View reviewed changes

fejta added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 16, 2018

leblancd force-pushed the kubeadm_dind_prow branch from c03d2fc to 890c284 Compare October 17, 2018 19:01

k8s-ci-robot reviewed Oct 17, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 17, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 17, 2018

k8s-ci-robot merged commit 9153828 into kubernetes:master Oct 17, 2018

Arvinderpal mentioned this pull request Feb 7, 2019

IPv6 Support kubernetes-sigs/kind#280

Closed

		@@ -0,0 +1,30 @@
		load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")

Add Kubeadm-DinD-Cluster based kubetest deployer for IPv6 CI #7529

Add Kubeadm-DinD-Cluster based kubetest deployer for IPv6 CI #7529

Conversation

leblancd commented Apr 3, 2018 • edited Loading

krzyzacy commented Apr 3, 2018

fejta commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented Apr 3, 2018

leblancd commented Apr 3, 2018

krzyzacy commented Apr 3, 2018

krzyzacy commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leblancd commented Oct 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krzyzacy commented Oct 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krzyzacy commented Oct 16, 2018

k8s-ci-robot left a comment

Choose a reason for hiding this comment

leblancd commented Oct 17, 2018

krzyzacy commented Oct 17, 2018

k8s-ci-robot left a comment

Choose a reason for hiding this comment

krzyzacy commented Oct 17, 2018

k8s-ci-robot commented Oct 17, 2018

k8s-ci-robot commented Oct 17, 2018

krzyzacy commented Oct 17, 2018

leblancd commented Oct 17, 2018

leblancd commented Apr 3, 2018 •

edited

Loading