Squashed 'release-tools/' changes from 5489de6..670bb0e

670bb0e Merge pull request kubernetes-csi#229 from marosset/fix-codespell-errors 35d5e78 Merge pull request kubernetes-csi#219 from yashsingh74/update-registry 63473cc Merge pull request kubernetes-csi#231 from coulof/bump-go-version-1.20.5 29a5c76 Merge pull request kubernetes-csi#228 from mowangdk/chore/adopt_kubernetes_recommand_labels 8dd2821 Update cloudbuild image with go 1.20.5 1df23db Merge pull request kubernetes-csi#230 from msau42/prow 1f92b7e Add ginkgo timeout to e2e tests to help catch any stuck tests 2b8b80e fixing some codespell errors c10b678 Merge pull request kubernetes-csi#227 from coulof/check-sidecar-supported-versions 72984ec chore: adopt kubernetes recommand label b055535 Header bd0a10b typo c39d73c Add comments f6491af Script to verify EOL sidecar version 4133d1d Merge pull request kubernetes-csi#226 from msau42/cloudbuild 8d519d2 Pin buildkit to v0.10.6 to workaround v0.11 bug with docker manifest 6e04a03 Merge pull request kubernetes-csi#224 from msau42/cloudbuild 26fdfff Update cloudbuild image 6613c39 Merge pull request kubernetes-csi#223 from sunnylovestiramisu/update 0e7ae99 Update k8s image repo url 77e47cc Merge pull request kubernetes-csi#222 from xinydev/fix-dep-version 155854b Fix dep version mismatch 8f83905 Merge pull request kubernetes-csi#221 from sunnylovestiramisu/go-update 1d3f94d Update go version to 1.20 to match k/k v1.27 e322ce5 Merge pull request kubernetes-csi#220 from andyzhangx/fix-golint-error b74a512 test: fix golint error 901bcb5 Update registry k8s.gcr.io -> registry.k8s.io aa61bfd Merge pull request kubernetes-csi#218 from xing-yang/update_csi_driver 7563d19 Update CSI_PROW_DRIVER_VERSION to v1.11.0 a2171be Merge pull request kubernetes-csi#216 from msau42/process cb98782 Merge pull request kubernetes-csi#217 from msau42/owners a11216e add new reviewers and remove inactive reviewers dd98675 Add step for checking builds b66c082 Merge pull request kubernetes-csi#214 from pohly/junit-fixes b9b6763 filter-junit.go: fix loss of testcases when parsing Ginkgo v2 JUnit d427783 filter-junit.go: preserve system error log 38e1146 prow.sh: publish individual JUnit files as separate artifacts 78c0fb7 Merge pull request kubernetes-csi#208 from jsafrane/skip-selinux 36e433e Skip SELinux tests in CI by default 348d4a9 Merge pull request kubernetes-csi#207 from RaunakShah/reviewers 1efc272 Merge pull request kubernetes-csi#206 from RaunakShah/update-prow 7d410d8 Changes to csi prow to run e2e tests in sidecars cfa5a75 Merge pull request kubernetes-csi#203 from humblec/test-vendor 4edd1d8 Add RaunakShah to CSI reviewers group 7ccc959 release tools update to 1.19 d24254f Merge pull request kubernetes-csi#202 from xing-yang/kind_0.14.0 0faa3fc Update to Kind v0.14.0 images ef4e1b2 Merge pull request kubernetes-csi#201 from xing-yang/add_1.24_image 4ddce25 Add 1.24 Kind image 7fe5149 Merge pull request kubernetes-csi#200 from pohly/bump-kubernetes-version 70915a8 prow.sh: update snapshotter version 31a3f38 Merge pull request kubernetes-csi#199 from pohly/bump-kubernetes-version 7577454 prow.sh: bump Kubernetes to v1.22.0 d29a2e7 Merge pull request kubernetes-csi#198 from pohly/csi-test-5.0.0 41cb70d prow.sh: sanity testing with csi-test v5.0.0 c85a63f Merge pull request kubernetes-csi#197 from pohly/fix-alpha-testing b86d8e9 support Kubernetes 1.25 + Ginkgo v2 ab0b0a3 Merge pull request kubernetes-csi#192 from andyzhangx/patch-1 7bbab24 Merge pull request kubernetes-csi#196 from humblec/non-alpha e51ff2c introduce control variable for non alpha feature gate configuration ca19ef5 Merge pull request kubernetes-csi#195 from pohly/fix-alpha-testing 3948331 fix testing with latest Kubernetes e4dab7f Merge pull request kubernetes-csi#194 from yselkowitz/registry-k8s-io 84a4d5a Move from k8s.gcr.io to registry.k8s.io 9a0260c fix boilerplate header 37d1104 Merge pull request kubernetes-csi#191 from pohly/go-1.18 db917f5 update to Go 1.18 335339f Merge pull request kubernetes-csi#187 from mauriciopoppe/remove-eol-windows-versions 890b87a Merge pull request kubernetes-csi#188 from pwschuurman/update-release-notes-docs 274bc9b Update Sidecar Release Process documentation to reference latest syntax for release-notes tool 87b6c37 Merge pull request kubernetes-csi#185 from Garima-Negi/fix-OWNERS-files f1de2c6 Fix OWNERS file - squashed commits 59ae38b Remove EOL windows versions from BUILD_PLATFORMS 5d66471 Merge pull request kubernetes-csi#186 from humblec/sp d066f1b Correct prow.sh typo and make codespell linter pass 762e22d Merge pull request kubernetes-csi#184 from pohly/image-publishing-troubleshooting 81e26c3 SIDECAR_RELEASE_PROCESS.md: add troubleshooting for image publishing 31aa44d Merge pull request kubernetes-csi#182 from chrishenzie/csi-sanity-version f49e141 Update csi-sanity test suite to v4.3.0 d9815c2 Merge pull request kubernetes-csi#181 from mauriciopoppe/armv7-support 05c1801 Add support to build arm/v7 images 4aedf35 Merge pull request kubernetes-csi#178 from xing-yang/timeout 2b9897e Increase build timeout 51d3702 Merge pull request kubernetes-csi#177 from mauriciopoppe/kind-image-1.23 a30efea Add kind image for 1.23 a6a1a79 Merge pull request kubernetes-csi#176 from pohly/go-1.17.3 0a2cf63 prow.sh: bump Go to 1.17.3 fc29fdd Merge pull request kubernetes-csi#141 from pohly/prune-replace-optional 5b9a1e0 Merge pull request kubernetes-csi#175 from jimdaga/patch-1 5eeb602 images: use k8s-staging-test-infra/gcb-docker-gcloud b46691a go-get-kubernetes.sh: make replace statement pruning optional git-subtree-dir: release-tools git-subtree-split: 670bb0e
mowangdk · Aug 6, 2023 · 06fd1a6 · 06fd1a6
1 parent 80b6445
commit 06fd1a6
Show file tree

Hide file tree

Showing 9 changed files with 402 additions and 97 deletions.
diff --git a/KUBERNETES_CSI_OWNERS_ALIASES b/KUBERNETES_CSI_OWNERS_ALIASES
@@ -22,24 +22,19 @@ aliases:
   - ggriffiths
   - gnufied
   - humblec
+  - mauriciopoppe
   - j-griffith
-  - Jiawei0227
   - jingxu97
   - jsafrane
   - pohly
+  - RaunakShah
+  - sunnylovestiramisu
   - xing-yang
 
 # This documents who previously contributed to Kubernetes-CSI
 # as approver.
-emeritus_approver:
+emeritus_approvers:
+- Jiawei0227
 - lpabon
 - sbezverk
 - vladimirvivien
-
-# This documents who previously contributed to Kubernetes-CSI
-# as reviewer.
-emeritus_reviewer:
-- lpabon
-- saad-ali
-- sbezverk
-- vladimirvivien
diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md
@@ -46,26 +46,38 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
 ## Release Process
 1. Identify all issues and ongoing PRs that should go into the release, and
   drive them to resolution.
-1. Download v2.8+ [K8s release notes
-  generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes)
+1. Download the latest version of the
+   [K8s release notes generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes)
+1. Create a
+   [Github personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
+   with `repo:public_repo` access
 1. Generate release notes for the release. Replace arguments with the relevant
-  information.
+   information.
     * Clean up old cached information (also needed if you are generating release
       notes for multiple repos)
       ```bash
       rm -rf /tmp/k8s-repo
       ```
     * For new minor releases on master:
         ```bash
-        GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
-        --github-org=kubernetes-csi --github-repo=external-provisioner
-        --required-author="" --output out.md
+        GITHUB_TOKEN=<token> release-notes \
+          --discover=mergebase-to-latest \
+          --org=kubernetes-csi \
+          --repo=external-provisioner \
+          --required-author="" \
+          --markdown-links \
+          --output out.md
         ```
     * For new patch releases on a release branch:
         ```bash
-        GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
-        --github-org=kubernetes-csi --github-repo=external-provisioner
-        --required-author="" --output out.md
+        GITHUB_TOKEN=<token> release-notes \
+          --discover=patch-to-latest \
+          --branch=release-1.1 \
+          --org=kubernetes-csi \
+          --repo=external-provisioner \
+          --required-author="" \
+          --markdown-links \
+          --output out.md
         ```
 1. Compare the generated output to the new commits for the release to check if
    any notable change missed a release note.
@@ -80,6 +92,8 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
 1. Check that all [canary CI
   jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
   and that test coverage is adequate for the changes that are going into the release.
+1. Check that the post-\<sidecar\>-push-images builds are succeeding.
+   [Example](https://k8s-testgrid.appspot.com/sig-storage-image-build#post-external-snapshotter-push-images)
 1. Make sure that no new PRs have merged in the meantime, and no PRs are in
    flight and soon to be merged.
 1. Create a new release following a previous release as a template. Be sure to select the correct
@@ -88,9 +102,9 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
 1. If release was a new major/minor version, create a new `release-<minor>`
    branch at that commit.
 1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
-1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
+1. Promote images from k8s-staging-sig-storage to registry.k8s.io/sig-storage. From
    the [k8s image
-   repo](https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage),
+   repo](https://github.com/kubernetes/k8s.io/tree/HEAD/registry.k8s.io/images/k8s-staging-sig-storage),
    run `./generate.sh > images.yaml`, and send a PR with the updated images.
    Once merged, the image promoter will copy the images from staging to prod.
 1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
@@ -100,6 +114,29 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
    and [k/k
    in-tree](https://github.com/kubernetes/kubernetes/tree/HEAD/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
 
+### Troubleshooting
+
+#### Image build jobs
+
+The following jobs are triggered after tagging to produce the corresponding
+image(s):
+https://k8s-testgrid.appspot.com/sig-storage-image-build
+
+Clicking on a failed build job opens that job in https://prow.k8s.io. Next to
+the job title is a rerun icon (circle with arrow). Clicking it opens a popup
+with a "rerun" button that maintainers with enough permissions can use. If in
+doubt, ask someone on #sig-release to rerun the job.
+
+Another way to rerun a job is to search for it in https://prow.k8s.io and click
+the rerun icon in the resulting job list:
+https://prow.k8s.io/?job=canary-csi-test-push-images
+
+#### Verify images
+
+Canary and staged images can be viewed at https://console.cloud.google.com/gcr/images/k8s-staging-sig-storage
+
+Promoted images can be viewed at https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/sig-storage
+
 ## Adding support for a new Kubernetes release
 
 1. Add the new release to `k8s_versions` in

diff --git a/build.make b/build.make
@@ -70,7 +70,7 @@ endif
 # Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables
 # to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below.
 
-# BUILD_PLATFORMS contains a set of tuples [os arch suffix base_image addon_image]
+# BUILD_PLATFORMS contains a set of tuples [os arch buildx_platform suffix base_image addon_image]
 # separated by semicolon. An empty variable or empty entry (= just a
 # semicolon) builds for the default platform of the current Go
 # toolchain.
@@ -85,20 +85,20 @@ FULL_LDFLAGS = $(LDFLAGS) $(IMPORTPATH_LDFLAGS) $(EXT_LDFLAGS)
 # defined by BUILD_PLATFORMS.
 $(CMDS:%=build-%): build-%: check-go-version-go
 	mkdir -p bin
-	# os_arch_seen captures all of the $$os-$$arch seen for the current binary
-	# that we want to build, if we've seen an $$os-$$arch before it means that
+	# os_arch_seen captures all of the $$os-$$arch-$$buildx_platform seen for the current binary
+	# that we want to build, if we've seen an $$os-$$arch-$$buildx_platform before it means that
 	# we don't need to build it again, this is done to avoid building
 	# the windows binary multiple times (see the default value of $$BUILD_PLATFORMS)
-	export os_arch_seen="" && echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
-		os_arch_seen_pre=$${os_arch_seen%%$$os-$$arch*}; \
+	export os_arch_seen="" && echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch buildx_platform suffix base_image addon_image; do \
+		os_arch_seen_pre=$${os_arch_seen%%$$os-$$arch-$$buildx_platform*}; \
 		if ! [ $${#os_arch_seen_pre} = $${#os_arch_seen} ]; then \
 			continue; \
 		fi; \
 		if ! (set -x; cd ./$(CMDS_DIR)/$* && CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '$(FULL_LDFLAGS)' -o "$(abspath ./bin)/$*$$suffix" .); then \
 			echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \
 			exit 1; \
 		fi; \
-		os_arch_seen+=";$$os-$$arch"; \
+		os_arch_seen+=";$$os-$$arch-$$buildx_platform"; \
 	done
 
 $(CMDS:%=container-%): container-%: build-%
@@ -148,22 +148,23 @@ DOCKER_BUILDX_CREATE_ARGS ?=
 $(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
 	set -ex; \
 	export DOCKER_CLI_EXPERIMENTAL=enabled; \
-	docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \
+	docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest --driver-opt image=moby/buildkit:v0.10.6; \
 	trap "docker buildx rm multiarchimage-buildertest" EXIT; \
 	dockerfile_linux=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile ]; then echo ./$(CMDS_DIR)/$*/Dockerfile; else echo Dockerfile; fi); \
 	dockerfile_windows=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile.Windows ]; then echo ./$(CMDS_DIR)/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
 	if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \
 	if ! [ -f "$$dockerfile_windows" ]; then \
-		build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe *[^ ]* *[^ ]*//g' -e 's/; *;/;/g' -e 's/;[ ]*$$//')"; \
+		build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *[^ ]* *.exe *[^ ]* *[^ ]*//g' -e 's/; *;/;/g' -e 's/;[ ]*$$//')"; \
 	fi; \
 	pushMultiArch () { \
 		tag=$$1; \
-		echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
+		echo "$$build_platforms" | tr ';' '\n' | while read -r os arch buildx_platform suffix base_image addon_image; do \
 			escaped_base_image=$${base_image/:/-}; \
+			escaped_buildx_platform=$${buildx_platform//\//-}; \
 			if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \
 			docker buildx build --push \
-				--tag $(IMAGE_NAME):$$arch-$$os-$$escaped_base_image$$tag \
-				--platform=$$os/$$arch \
+				--tag $(IMAGE_NAME):$$escaped_buildx_platform-$$os-$$escaped_base_image$$tag \
+				--platform=$$os/$$buildx_platform \
 				--file $$(eval echo \$${dockerfile_$$os}) \
 				--build-arg binary=./bin/$*$$suffix \
 				--build-arg ARCH=$$arch \
@@ -172,13 +173,14 @@ $(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
 				--label revision=$(REV) \
 				.; \
 		done; \
-		images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
+		images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch buildx_platform suffix base_image addon_image; do \
 			escaped_base_image=$${base_image/:/-}; \
+			escaped_buildx_platform=$${buildx_platform//\//-}; \
 			if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \
-			echo $(IMAGE_NAME):$$arch-$$os-$$escaped_base_image$$tag; \
+			echo $(IMAGE_NAME):$$escaped_buildx_platform-$$os-$$escaped_base_image$$tag; \
 		done); \
 		docker manifest create --amend $(IMAGE_NAME):$$tag $$images; \
-		echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
+		echo "$$build_platforms" | tr ';' '\n' | while read -r os arch buildx_platform suffix base_image addon_image; do \
 			if [ $$os = "windows" ]; then \
 				escaped_base_image=$${base_image/:/-}; \
 				if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \

diff --git a/cloudbuild.yaml b/cloudbuild.yaml
@@ -13,12 +13,11 @@
 # See https://github.com/kubernetes/test-infra/blob/HEAD/config/jobs/image-pushing/README.md
 # for more details on image pushing process in Kubernetes.
 #
-# To promote release images, see https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage.
+# To promote release images, see https://github.com/kubernetes/k8s.io/tree/HEAD/registry.k8s.io/images/k8s-staging-sig-storage.
 
 # This must be specified in seconds. If omitted, defaults to 600s (10 mins).
-# Building three images in external-snapshotter takes roughly half an hour,
-# sometimes more.
-timeout: 3600s
+# Building three images in external-snapshotter takes more than an hour.
+timeout: 7200s
 # This prevents errors if you don't use both _GIT_TAG and _PULL_BASE_REF,
 # or any new substitutions added in the future.
 options:
@@ -27,7 +26,7 @@ steps:
   # The image must contain bash and curl. Ideally it should also contain
   # the desired version of Go (currently defined in release-tools/prow.sh),
   # but that just speeds up the build and is not required.
-  - name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20210331-c732583'
+  - name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20230623-56e06d7c18'
     entrypoint: ./.cloudbuild.sh
     env:
     - GIT_TAG=${_GIT_TAG}