gce: add scripts for CI test and nightly builds #445

cpanato · 2020-12-03T17:15:24Z

The first part to add GCE image builder to run on CI

This PR adds the script that will be used in prow to build the images for CAPI/GCE

Also adds scripts to run nightly builds

Fixes: #437

Will open the test-infra PR to add the job

Please let me know if there is a better way to use boskos

/assign @detiber @vincepri

cpanato · 2020-12-03T17:19:15Z

Also, do we need to create any accounts?

cpanato · 2020-12-03T17:20:11Z

/hold for discussion and pending items

detiber · 2020-12-03T19:15:25Z

Also, do we need to create any accounts?

Shouldn't we should be able to leverage existing accounts similar to what we are doing with CAPG :)

detiber

It might be good to dedupe boskos management approaches with https://github.com/kubernetes-sigs/image-builder/pull/423/files#diff-1c56e5a5df7b6f697f270ac094d292b4a3929e1ebafd0a1970f35c895d4a986e

images/capi/hack/checkout_account.py

images/capi/hack/checkin_account.py

images/capi/hack/checkout_account.py

images/capi/hack/heartbeat_account.py

images/capi/packer/gce/scripts/ci-gce.sh

images/capi/hack/boskos.py

images/capi/packer/gce/scripts/ci-gce.sh

images/capi/hack/boskos.py

detiber

A few suggestions around the exception handling, otherwise this is looking pretty good

images/capi/hack/boskos.py

detiber · 2021-03-08T18:58:54Z

/lgtm

CecileRobertMichon · 2021-03-09T22:49:01Z

Is there an associated test-infra PR to add the job as optional so we can use it to test the PR before merging?

codenrhoden

I think everything I have is a nit, but it would be nice to see the typo fixed first.

images/capi/packer/gce/ci/nightly/overwrite-1-18.json

images/capi/packer/gce/ci/nightly/overwrite-1-19.json

images/capi/packer/gce/ci/nightly/overwrite-1-20.json

codenrhoden · 2021-05-04T16:29:38Z

images/capi/scripts/ci-gce-nightly.sh

+
+# build image for 1.18
+# using PACKER_FLAGS=-force to overwrite the previous image and keep the same name
+su - packer -c "bash -c 'cd /home/prow/go/src/sigs.k8s.io/image-builder/images/capi && PATH=$PATH:~packer/.local/bin:/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin GCP_PROJECT_ID=$GCP_PROJECT GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS PACKER_VAR_FILES=packer/gce/ci/nightly/overwrite-1-18.json PACKER_FLAGS=-force make deps-gce build-gce-all'"


deps-gce is already a dependency of the gce-all build target, so adding that explicitly ins't strictly necessary. But I have no problem with it being listed out.

i will keep that just for visibility

codenrhoden · 2021-05-04T16:33:37Z

images/capi/scripts/ci-gce-nightly.sh

+su - packer -c "bash -c 'cd /home/prow/go/src/sigs.k8s.io/image-builder/images/capi && PATH=$PATH:~packer/.local/bin:/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin GCP_PROJECT_ID=$GCP_PROJECT GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS PACKER_VAR_FILES=packer/gce/ci/nightly/overwrite-1-20.json PACKER_FLAGS=-force make deps-gce build-gce-all'"
+
+echo "Displaying the generated image information"
+filter="name~cluster-api-ubuntu-1804-*"


I notice this filter includes 1804 only, but the ci-gce script does not (just has cluster-api-ubuntu-*. Do we want to make this one the same so that it picks up 2004? I see that 20.04 Ubuntu builds are currently commented out in the Makefile (however, not completely because it still shows up in make help), but if we change this filter I think it would mean that you wouldn't need to change anything when Ubuntu 20.04 does get reinstated?

good catch! thanks!

thanks for your review and feedback @codenrhoden
made the changes requested

codenrhoden · 2021-05-10T16:41:32Z

/test pull-azure-vhds
/lgtm

That windows Azure failure (I want to make some changes soon that are a little more sane about what tests run depending on what changed - there's no need for Azure tests in this case for example) doesn't quite look like a flake to me, but we'll see what happens.

spiffxp · 2021-05-10T19:51:27Z

Have you all looked into https://cloud.google.com/build/docs/building/build-vm-images-with-packer at all? My glance of the CI scripts is that they boil down to packer calls

https://cloud.google.com/build/docs/building/build-vm-images-with-packer#required_iam_permissions is something we would want scripted for the projects that need this

https://cloud.google.com/build/docs/building/build-vm-images-with-packer#creating_a_packer_builder_image is something we might want to push to k8s-staging-test-infra or something

cpanato · 2021-05-11T08:43:56Z

@spiffxp to be honest this is the first time I'm seeing this, looks very nice.

I tried the scripts using my personal GCP account and everything worked fine, and regarding the API I just enable the compute one.

I think this looks interesting, can we do that in a follow up maybe? I think we need to change some things in the image builder to make this works
but if you think this should be the approach for now, I can refactor everything, I don't have strong opinions here.

cpanato · 2021-05-16T08:07:57Z

/test pull-image-builder-gcp-all

cpanato · 2021-05-16T08:28:20Z

I think we can override the azure-vhds tests and investigate that in a follow you because that is not part of this PR

WDYT @codenrhoden ?

and to close this PR need some final reviews/approval:
/assign @vincepri @detiber @CecileRobertMichon

spiffxp · 2021-05-16T22:25:52Z

I am ok merging this to unblock and finding an alternate approach as followup

spiffxp · 2021-05-16T22:26:22Z

I'll review more closely at keyboard tomorrow

cpanato · 2021-05-17T07:02:46Z

I am ok merging this to unblock and finding an alternate approach as followup

I'm checking what are the changes need to apply here to use cloudbuild together with packer to build the image

cpanato · 2021-05-17T07:46:48Z

open a 🧵 to discuss better the cloudbuild option to build CAPG GCE images: https://kubernetes.slack.com/archives/CCK68P2Q2/p1621237438253400

codenrhoden · 2021-05-18T16:14:28Z

/lgtm
/approve

Agreed that merging this now and exploring other options as follow-up makes sense. Will see what happens with the flaky Azure test.

k8s-ci-robot · 2021-05-18T16:14:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: codenrhoden, cpanato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~images/capi/OWNERS~~ [codenrhoden]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

CecileRobertMichon · 2021-05-18T16:33:50Z

The error in the failed pull-azure-vhds job above is:

�[0;32m    vhd-windows-2019: fatal: [default]: FAILED! => {"changed": true, "cmd": "$exists=docker network ls -f name=host -q\nif (-not $exists) { docker network create -d nat host }", "delta": "0:00:01.455251", "end": "2021-05-16 08:12:06.223301", "msg": "non-zero return code", "rc": 1, "start": "2021-05-16 08:12:04.768050", "stderr": "error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks?filters=%7B%22name%22%3A%7B%22host%22%3Atrue%7D%7D: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.\nerror during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks/create: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.", "stderr_lines": ["error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks?filters=%7B%22name%22%3A%7B%22host%22%3Atrue%7D%7D: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.", "error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks/create: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running."], "stdout": "", "stdout_lines": []}�[0m

@jsturtevant @perithompson have you seen this before?

cpanato · 2021-05-19T07:58:10Z

will override this failure

cpanato · 2021-05-19T07:58:20Z

/override pull-azure-vhds

cpanato · 2021-05-19T07:59:35Z

I cant :/

/test pull-azure-vhds

perithompson · 2021-05-19T11:55:08Z

The error in the failed pull-azure-vhds job above is:

�[0;32m    vhd-windows-2019: fatal: [default]: FAILED! => {"changed": true, "cmd": "$exists=docker network ls -f name=host -q\nif (-not $exists) { docker network create -d nat host }", "delta": "0:00:01.455251", "end": "2021-05-16 08:12:06.223301", "msg": "non-zero return code", "rc": 1, "start": "2021-05-16 08:12:04.768050", "stderr": "error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks?filters=%7B%22name%22%3A%7B%22host%22%3Atrue%7D%7D: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.\nerror during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks/create: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.", "stderr_lines": ["error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks?filters=%7B%22name%22%3A%7B%22host%22%3Atrue%7D%7D: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.", "error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.40/networks/create: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running."], "stdout": "", "stdout_lines": []}�[0m

@jsturtevant @perithompson have you seen this before?

No this is a new one on me, looks like the docker service didn't start on this node?

k8s-ci-robot assigned detiber and vincepri Dec 3, 2020

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 3, 2020

k8s-ci-robot requested review from davidewatson and ncdc December 3, 2020 17:15

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 3, 2020

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2020

detiber reviewed Dec 3, 2020

View reviewed changes

cpanato force-pushed the GH-437 branch 2 times, most recently from fc17458 to f281516 Compare December 5, 2020 10:15

detiber reviewed Dec 8, 2020

View reviewed changes

cpanato force-pushed the GH-437 branch from f281516 to bdb8139 Compare December 10, 2020 13:44

detiber reviewed Dec 10, 2020

View reviewed changes

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

cpanato force-pushed the GH-437 branch 3 times, most recently from 788009b to d2e19aa Compare December 12, 2020 14:30

cpanato requested a review from detiber December 15, 2020 13:23

kkeshavamurthy reviewed Dec 15, 2020

View reviewed changes

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

kkeshavamurthy reviewed Dec 15, 2020

View reviewed changes

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

cpanato force-pushed the GH-437 branch from d2e19aa to 42a28ed Compare December 16, 2020 14:52

kkeshavamurthy reviewed Dec 16, 2020

View reviewed changes

images/capi/hack/boskos.py Outdated Show resolved Hide resolved

cpanato force-pushed the GH-437 branch 3 times, most recently from a4a5f5e to 5b3fa6a Compare February 11, 2021 15:49

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2021

dims mentioned this pull request May 2, 2021

switching on compute api and other APIs in k8s-staging-cluster-api-gcp kubernetes/k8s.io#1993

Closed

codenrhoden reviewed May 4, 2021

View reviewed changes

cpanato force-pushed the GH-437 branch from 80fd431 to 0f25f5d Compare May 10, 2021 08:12

cpanato mentioned this pull request May 10, 2021

capg: enable compute api for CAPG to build the node images kubernetes/k8s.io#2025

Merged

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 10, 2021

cpanato added 2 commits May 16, 2021 09:54

gce: add scripts for CI test

792f618

gce: add scripts for CI nightly builds

471099d

cpanato force-pushed the GH-437 branch from 0f25f5d to 471099d Compare May 16, 2021 07:54

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 16, 2021

cpanato mentioned this pull request May 18, 2021

Figure out how to publish images kubernetes-sigs/cluster-api-provider-gcp#152

Open

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 18, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 18, 2021

k8s-ci-robot merged commit 517f780 into kubernetes-sigs:master May 19, 2021

cpanato deleted the GH-437 branch May 19, 2021 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gce: add scripts for CI test and nightly builds #445

gce: add scripts for CI test and nightly builds #445

cpanato commented Dec 3, 2020 •

edited

Loading

cpanato commented Dec 3, 2020

cpanato commented Dec 3, 2020

detiber commented Dec 3, 2020

detiber left a comment

detiber left a comment

detiber commented Mar 8, 2021

CecileRobertMichon commented Mar 9, 2021

codenrhoden left a comment

codenrhoden May 4, 2021

cpanato May 10, 2021

codenrhoden May 4, 2021

cpanato May 10, 2021

cpanato May 10, 2021

codenrhoden commented May 10, 2021

spiffxp commented May 10, 2021

cpanato commented May 11, 2021

cpanato commented May 16, 2021

cpanato commented May 16, 2021

spiffxp commented May 16, 2021

spiffxp commented May 16, 2021

cpanato commented May 17, 2021

cpanato commented May 17, 2021

codenrhoden commented May 18, 2021

k8s-ci-robot commented May 18, 2021

CecileRobertMichon commented May 18, 2021

cpanato commented May 19, 2021

cpanato commented May 19, 2021

cpanato commented May 19, 2021

perithompson commented May 19, 2021

gce: add scripts for CI test and nightly builds #445

gce: add scripts for CI test and nightly builds #445

Conversation

cpanato commented Dec 3, 2020 • edited Loading

cpanato commented Dec 3, 2020

cpanato commented Dec 3, 2020

detiber commented Dec 3, 2020

detiber left a comment

Choose a reason for hiding this comment

detiber left a comment

Choose a reason for hiding this comment

detiber commented Mar 8, 2021

CecileRobertMichon commented Mar 9, 2021

codenrhoden left a comment

Choose a reason for hiding this comment

codenrhoden May 4, 2021

Choose a reason for hiding this comment

cpanato May 10, 2021

Choose a reason for hiding this comment

codenrhoden May 4, 2021

Choose a reason for hiding this comment

cpanato May 10, 2021

Choose a reason for hiding this comment

cpanato May 10, 2021

Choose a reason for hiding this comment

codenrhoden commented May 10, 2021

spiffxp commented May 10, 2021

cpanato commented May 11, 2021

cpanato commented May 16, 2021

cpanato commented May 16, 2021

spiffxp commented May 16, 2021

spiffxp commented May 16, 2021

cpanato commented May 17, 2021

cpanato commented May 17, 2021

codenrhoden commented May 18, 2021

k8s-ci-robot commented May 18, 2021

CecileRobertMichon commented May 18, 2021

cpanato commented May 19, 2021

cpanato commented May 19, 2021

cpanato commented May 19, 2021

perithompson commented May 19, 2021

cpanato commented Dec 3, 2020 •

edited

Loading