Add support for building multi-arch images #3141

tstapler · 2022-07-27T00:53:06Z

I took a prototype by @bobbypage and added support for additional architectures, simplified the docker build, and wrote a script to test the images generated by release.sh.

Note: My GCR repository is private you'll have to use your own container registry that you have permissions to.

VERSION=v0.44.1-test-8 IMAGE_NAME=gcr.io/tstapler-gke-dev/cadvisor make release

Starting each of the images and curling the healthz endpoint:

build/check_container.sh gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8
Testing that we can run gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8 on ppc64le and curl the /healthz endpoint

Unable to find image 'gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8' locally
v0.44.1-test-8: Pulling from tstapler-gke-dev/cadvisor
602896d44249: Already exists
e803f340c0e7: Pulling fs layer
68310738913e: Pulling fs layer
4f4fb700ef54: Pulling fs layer
cab919d5ac51: Pulling fs layer
cab919d5ac51: Waiting
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
68310738913e: Verifying Checksum
68310738913e: Download complete
e803f340c0e7: Verifying Checksum
e803f340c0e7: Download complete
cab919d5ac51: Verifying Checksum
cab919d5ac51: Download complete
e803f340c0e7: Pull complete
68310738913e: Pull complete
4f4fb700ef54: Pull complete
cab919d5ac51: Pull complete
Digest: sha256:00a2e625014c262f7cef381451cebad48d46908549f1f508797865bbaf072aaa
Status: Downloaded newer image for gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8

Testing the container with curl:
ok

Success!

Cleaning up the container 9f6aebe538c2775030b0c31d7bb875b0f00b54d913c28898ac7ad2d1f36e2694
9f6aebe538c2775030b0c31d7bb875b0f00b54d913c28898ac7ad2d1f36e2694
Untagged: gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8
Untagged: gcr.io/tstapler-gke-dev/cadvisor@sha256:00a2e625014c262f7cef381451cebad48d46908549f1f508797865bbaf072aaa
Deleted: sha256:c15692c504860fe061c739af8d7c1113b5f3c90b738b1fbd5ce7f1526eb58bdf
Deleted: sha256:f9bdfbb0c703807ba7314b93da1ca15b1acebf9c950c08c5699ff693992d42ae
Deleted: sha256:49d7509cfbf1e63e49f864238813d32a56f7a768885f9a01a99634c90238007b
Deleted: sha256:5ad517b5d81b6af6b0b88a8d9c604b7d0454deaaf674584e74b88dd3f287f59b
Deleted: sha256:1b228a3ba7f23b82e9e9f8f07bfea8caf7c8e6cb9a35097fe325592b53f6ad61

Testing that we can run gcr.io/tstapler-gke-dev/cadvisor:v0.44.1-test-8 on arm and curl the /healthz endpoint
...

Co-authored-by: David Porter porterdavid@google.com
Signed-of-by: Tyler Stapler tystapler@gmail.com

k8s-ci-robot · 2022-07-27T00:53:15Z

Hi @tstapler. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tstapler · 2022-07-27T00:54:33Z

deploy/Dockerfile.ppc64le

@@ -1,5 +1,7 @@
 FROM ppc64le/alpine:3.15
 MAINTAINER dashpole@google.com lysannef@us.ibm.com
+# Deprecated: the Dockerfile in this directory should support ppc64le


We should be able to remove this file now. I added the deprecation message to be cautious about breaking people's workflow.

ack, sounds good!

tstapler · 2022-07-27T00:55:29Z

deploy/Dockerfile

+# ipmctl only supports Intel x86_64 processors.
+# https://github.com/intel/ipmctl/issues/163
+RUN if [ "$(uname --machine)" = "x86_64" ]; then \
+    git clone -b v02.00.00.3885 https://github.com/intel/ipmctl/ && \


I had to update from 3820 to 3885 to fix a compiler warning which caused errors intel/ipmctl#169

tstapler · 2022-07-27T00:57:34Z

deploy/Dockerfile


-RUN apk --no-cache add libc6-compat device-mapper findutils zfs build-base linux-headers python3 bash git wget cmake pkgconfig ndctl-dev && \
+# Install build depdencies for all supported arches
+RUN apk --no-cache add bash build-base cmake device-mapper findutils git \


The dependencies should be identical, I just alphabetically

thanks for cleanup!

tstapler · 2022-07-27T00:58:10Z

build/release.sh

-echo "Release info (copy to the release page):"
-echo
-echo "Docker Image: N/A"
-echo "gcr.io Image: $gcr_tag"


Let me know if I should build a similar release info page

That would be great, to make it simpler to cut releases.

If we can just include the gcr tag, hash, and hashes of the cadvisor binaries, it's all we need

Alright the latest version should print out a summary. I updated the releasing.md file to reflect the changes.

Makefile

bobbypage · 2022-07-27T21:03:57Z

/ok-to-test

bobbypage · 2022-07-27T21:25:25Z

build/build.sh

@@ -16,11 +16,15 @@

 set -e

+declare -A arches=( ["amd64"]="x86_64" ["arm"]="armv7l" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" )


As discussed offline, let's just skip to amd64 armv7 and arm64 since these are the most popular ones and we can try to test.

Updated this to only include the architectures listed above.

bobbypage · 2022-07-27T21:26:49Z

build/release.sh

+  arch_specific_image="${image_name}-${arch}:${VERSION}"
+  docker buildx build --platform "linux/${arch}" -f deploy/Dockerfile -t "$arch_specific_image"  --progress plain --push .
+  docker manifest create --amend "$final_image" "$arch_specific_image"
+  docker manifest annotate --os=linux --arch="$arch" "$final_image" "$arch_specific_image"


just curious, for my own understanding ,why do we need to use docker manifest annotate here? Does docker buildx not do this automatically?

docker buildx build will create a multi-arch manifest if you specify multiple platforms --platform linux/amd64,linux/arm64 and the --push flag.

However, it does not tag intermediate containers like cadvisor-arm64. It makes it a little harder when you want to test a specific architecture locally.

ah gotcha, thanks for the explanation!

bobbypage · 2022-07-27T21:29:15Z

Thank you @tstapler for all your work on this!

Please update https://github.com/google/cadvisor/blob/master/docs/development/releasing.md with the new details (since make release now builds the docker image automatically). Thank you!

bobbypage · 2022-07-27T21:44:09Z

The pull-cadvisor-e2e is failing because it's using go1.19rc2 in the presubmit, but go 1.19 isn't submitted yet, so we can ignore that for now

tstapler · 2022-07-28T17:50:35Z

deploy/Dockerfile

+ADD go.mod go.sum ./
+RUN go mod download
+ADD cmd/go.mod cmd/go.sum ./cmd/
+RUN cd cmd && go mod download


Added caching of the go deps so that if you want to repeatedly build the docker container locally it will be pretty quick.

tstapler · 2022-07-28T17:53:00Z

build/release.sh

-unset GO_FLAGS
-build/build.sh
+docker buildx inspect cadvisor-builder > /dev/null \
+|| docker buildx create --name cadvisor-builder --use


I made this builder persistent, it helps with local testing because the unchanged parts of the image are cached. Let me know if you want me to return to deleting the builder afterwards.

sounds good!

tstapler · 2022-07-28T17:53:26Z

build/release.sh

+  fi
+fi
+
+read -p "Please confirm: $VERSION is the desired version (Type y/n to continue):" -n 1 -r


Instead of having the user confirm the version at the end, I added a prompt here.

tstapler · 2022-07-28T17:55:30Z

docs/development/releasing.md

 ```

-## 5. Cut the release
+## 4. Cut the release


Should I add running check_container.sh as a step to confirm that you can start all of the containers that were generated?

That's a good idea, let's add it as validation step

Added the validation step

tstapler · 2022-07-28T17:56:27Z

build/release.sh

+declare -A arches=( ["amd64"]="x86_64" ["arm"]="arm" ["arm64"]="aarch64")
+
+for arch in "${arches[@]}"; do
+  if ! hash "qemu-${arch}-static"; then


Added a check to see if qemu was installed to prevent people being unable to release and not understanding why.

Thanks for adding this check, will help alot to debug

bobbypage · 2022-07-29T00:14:40Z

LGTM!

bobbypage · 2022-07-29T00:15:56Z

Thanks @tstapler for working on this and all of the cleanups as well!

This fixes long standing issue #2763

Only install libipmctl for amd64 versions of the docker image. Updated ipmctl to v02.00.00.3885 to fix intel/ipmctl#169 Updated base image to alpine:1.16 from alpine:1.15 to add zfs support on "arm" architecture. Co-authored-by: David Porter <porterdavid@google.com> Signed-of-by: Tyler Stapler <tystapler@gmail.com>

tstapler · 2022-07-29T16:41:33Z

I think I've addressed all comments now.

k8s-ci-robot · 2022-07-29T16:47:25Z

@tstapler: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cadvisor-e2e	`56ac6ef`	link	true	`/test pull-cadvisor-e2e`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

bobbypage · 2022-07-29T20:20:00Z

LGTM!

Add support for building multi-arch images [#3141](google/cadvisor#3141) - cAdvisor v0.45.0 now has a multi arch image

k8s-ci-robot added the needs-ok-to-test label Jul 27, 2022

tstapler commented Jul 27, 2022

View reviewed changes

Makefile Show resolved Hide resolved

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Jul 27, 2022

bobbypage reviewed Jul 27, 2022

View reviewed changes

tstapler force-pushed the multi-arch-builds branch 4 times, most recently from dbb963e to 8d763d1 Compare July 28, 2022 17:41

tstapler commented Jul 28, 2022

View reviewed changes

tstapler requested a review from bobbypage July 28, 2022 17:54

tstapler commented Jul 28, 2022

View reviewed changes

tstapler force-pushed the multi-arch-builds branch 2 times, most recently from a30ca19 to 612866b Compare July 28, 2022 23:01

tstapler force-pushed the multi-arch-builds branch from 612866b to 56ac6ef Compare July 29, 2022 16:42

bobbypage merged commit 98b8f4a into google:master Jul 29, 2022

bobbypage mentioned this pull request Aug 3, 2022

support for apple silicon linux/arm64/v8 platform #2763

Closed

94rain mentioned this pull request Oct 20, 2022

Does not work with Macbook M1 Docker for mac. #3187

Open

Logicer16 mentioned this pull request Oct 20, 2022

Cadvisor for ARM #1236

Closed

philyuchkoff added a commit to philyuchkoff/dockermon that referenced this pull request Jan 20, 2023

some fix

95a5651

Add support for building multi-arch images [#3141](google/cadvisor#3141) - cAdvisor v0.45.0 now has a multi arch image

th3noname mentioned this pull request Apr 12, 2023

cadvisor:latest doesn't provide arm support hetznercloud/apps#98

Closed

bboreham mentioned this pull request Oct 16, 2023

Default image on gcr.io downloads a 3 year old image #3363

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for building multi-arch images #3141

Add support for building multi-arch images #3141

tstapler commented Jul 27, 2022

k8s-ci-robot commented Jul 27, 2022

tstapler Jul 27, 2022

bobbypage Jul 27, 2022

tstapler Jul 27, 2022

tstapler Jul 27, 2022

bobbypage Jul 27, 2022

tstapler Jul 27, 2022

bobbypage Jul 27, 2022

tstapler Jul 27, 2022

bobbypage commented Jul 27, 2022

bobbypage Jul 27, 2022

tstapler Jul 28, 2022

bobbypage Jul 27, 2022

tstapler Jul 27, 2022

bobbypage Jul 27, 2022

bobbypage commented Jul 27, 2022 •

edited

Loading

bobbypage commented Jul 27, 2022 •

edited

Loading

tstapler Jul 28, 2022

tstapler Jul 28, 2022

bobbypage Jul 29, 2022

tstapler Jul 28, 2022

tstapler Jul 28, 2022

bobbypage Jul 29, 2022

tstapler Jul 29, 2022

tstapler Jul 28, 2022

bobbypage Jul 29, 2022

bobbypage commented Jul 29, 2022

bobbypage commented Jul 29, 2022 •

edited

Loading

tstapler commented Jul 29, 2022

k8s-ci-robot commented Jul 29, 2022

bobbypage commented Jul 29, 2022

		@@ -16,11 +16,15 @@

		set -e

		declare -A arches=( ["amd64"]="x86_64" ["arm"]="armv7l" ["arm64"]="aarch64" ["ppc64le"]="ppc64le" ["s390x"]="s390x" )

Add support for building multi-arch images #3141

Add support for building multi-arch images #3141

Conversation

tstapler commented Jul 27, 2022

k8s-ci-robot commented Jul 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage commented Jul 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage commented Jul 27, 2022 • edited Loading

bobbypage commented Jul 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbypage commented Jul 29, 2022

bobbypage commented Jul 29, 2022 • edited Loading

tstapler commented Jul 29, 2022

k8s-ci-robot commented Jul 29, 2022

bobbypage commented Jul 29, 2022

bobbypage commented Jul 27, 2022 •

edited

Loading

bobbypage commented Jul 27, 2022 •

edited

Loading

bobbypage commented Jul 29, 2022 •

edited

Loading