Bump kubernetes 1.28 #751

wyike · 2023-08-23T02:27:26Z

What this PR does / why we need it:

Bump kubernetes dependencies to 1 .28

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #750

Special notes for your reviewer:

Manually test (3 tests) done on CAPI1.5 + CAPV1.8 + ubuntu-2004-kube-v1.28.0
Will have follow up PR to bump manifest and charts versions

Release note:

Bump kubernetes dependencies to 1 .28

k8s-ci-robot · 2023-08-23T02:27:34Z

Welcome @wyike!

It looks like this is your first PR to kubernetes/cloud-provider-vsphere 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-vsphere has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2023-08-23T02:27:35Z

Hi @wyike. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wyike · 2023-08-23T02:34:16Z

cmd/vsphere-cloud-controller-manager/main.go

@@ -80,7 +81,7 @@ func main() {
 	}

 	fs := command.Flags()
-	namedFlagSets := ccmOptions.Flags(app.ControllerNames(app.DefaultInitFuncConstructors), app.ControllersDisabledByDefault.List(), app.AllWebhooks, app.DisabledByDefaultWebhooks)
+	namedFlagSets := ccmOptions.Flags(app.ControllerNames(app.DefaultInitFuncConstructors), app.ControllersDisabledByDefault.List(), names.CCMControllerAliases(), app.AllWebhooks, app.DisabledByDefaultWebhooks)


New KCM and CCM change on 1.28: kubernetes/kubernetes#115813

wyike · 2023-08-23T02:38:37Z

test/e2e/go.mod

@@ -166,15 +169,15 @@ require (
 	k8s.io/cli-runtime v0.27.2 // indirect


I didn't bump the indirect dependencies( line 167-171, 173(k8s), and 176(controllerruntime) ) to 1.28 related.
They seem indirectly used by cluster api and helm. What's your opinion?

Yup should be fine

wyike · 2023-08-23T02:43:53Z

/assign @lubronzhan @sbueringer

lubronzhan · 2023-08-23T03:05:02Z

Could you create a new release file like https://github.com/kubernetes/cloud-provider-vsphere/tree/master/releases/v1.27

Also add helm chart changes, you can just do following steps
https://github.com/kubernetes/cloud-provider-vsphere/blob/master/charts/vsphere-cpi/README.md?plain=1#L148-L161

wyike · 2023-08-23T03:10:07Z

Could you create a new release file like https://github.com/kubernetes/cloud-provider-vsphere/tree/master/releases/v1.27

Also add helm chart changes, you can just do following steps https://github.com/kubernetes/cloud-provider-vsphere/blob/master/charts/vsphere-cpi/README.md?plain=1#L148-L161

Yes, planned to do in a follow up PR.

Do you want me to include it to this PR together?

lubronzhan · 2023-08-23T03:23:13Z

Ok you can do it in follow up PR

lubronzhan · 2023-08-23T03:23:25Z

/ok-to-test

wyike · 2023-08-23T07:46:17Z

test/e2e/go.mod

@@ -163,18 +165,18 @@ require (
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	k8s.io/apiextensions-apiserver v0.27.2 // indirect
 	k8s.io/apiserver v0.27.2 // indirect
-	k8s.io/cli-runtime v0.27.2 // indirect
+	k8s.io/cli-runtime v0.28.0 // indirect


I still have bumped the indirect k8s.io/kubectl to 0.28.0 and several other indirect dependencies are updated automatically. Otherwise go test e2e_suite_test.go will have compile error:

go test e2e_suite_test.go # k8s.io/cli-runtime/pkg/resource ../../../../../../pkg/mod/k8s.io/cli-runtime@v0.27.2/pkg/resource/query_param_verifier.go:83:38: cannot use oapi (variable of type *"github.com/google/gnostic-models/openapiv2".Document) as *"github.com/google/gnostic/openapiv2".Document value in argument to supportsQueryParam ../../../../../../pkg/mod/k8s.io/cli-runtime@v0.27.2/pkg/resource/query_param_verifier.go:86:36: cannot use oapi (variable of type *"github.com/google/gnostic-models/openapiv2".Document) as *"github.com/google/gnostic/openapiv2".Document value in argument to supportsQueryParam # k8s.io/kubectl/pkg/util/openapi ../../../../../../pkg/mod/k8s.io/kubectl@v0.27.2/pkg/util/openapi/openapi_getter.go:36:42: cannot use &CachedOpenAPIGetter{} (value of type *CachedOpenAPIGetter) as discovery.OpenAPISchemaInterface value in variable declaration: *CachedOpenAPIGetter does not implement discovery.OpenAPISchemaInterface (wrong type for method OpenAPISchema) have OpenAPISchema() (*"github.com/google/gnostic/openapiv2".Document, error) want OpenAPISchema() (*"github.com/google/gnostic-models/openapiv2".Document, error) ../../../../../../pkg/mod/k8s.io/kubectl@v0.27.2/pkg/util/openapi/openapi_getter.go:49:28: cannot use g.openAPIClient.OpenAPISchema() (value of type *"github.com/google/gnostic-models/openapiv2".Document) as *"github.com/google/gnostic/openapiv2".Document value in assignment ../../../../../../pkg/mod/k8s.io/kubectl@v0.27.2/pkg/util/openapi/openapi_getter.go:78:46: cannot use oapi (variable of type *"github.com/google/gnostic-models/openapiv2".Document) as *"github.com/google/gnostic/openapiv2".Document value in argument to NewOpenAPIData ../../../../../../pkg/mod/k8s.io/kubectl@v0.27.2/pkg/util/openapi/openapi.go:52:38: cannot use doc (variable of type *"github.com/google/gnostic/openapiv2".Document) as *"github.com/google/gnostic-models/openapiv2".Document value in argument to proto.NewOpenAPIData FAIL command-line-arguments [build failed]

I didn't bump other k8s.io/xx or sigs.k8s.io/controller-runtime because bump either will throw other incompatible compile errors.
We can bump capi to 1.6 and sigs.k8s.io/controller-runtime to 0.16, along with other k8s.io/xxx to have good compatibility in future.

We can probably bump to a version of CAPI main once the CR bump PR there merges (probably today). Just like we used a v1.5 pre-release version (l.185-188)

sbueringer · 2023-08-23T08:09:31Z

In general looks fine to me. You just might want to bump to a CAPI main version in a bit. I expect the CR bump there to merge in the next 1-3 hours

sbueringer · 2023-08-23T11:37:21Z

CR bump is merged in CAPI: kubernetes-sigs/cluster-api#8999

wyike · 2023-08-23T11:59:27Z

CR bump is merged in CAPI: kubernetes-sigs/cluster-api#8999

Thanks @sbueringer

I tried to point the capi to sigs.k8s.io/cluster-api v1.5.0-rc.0.0.20230823105256-932cdc4f95cf and upgrade cr to 0.16. It's not very smooth:

go.opentelemetry.io/otel/metric version (v0.31.0) seems a little old in capi.

go mod tidy
go: finding module for package go.opentelemetry.io/otel/metric/instrument/syncint64
go: finding module for package go.opentelemetry.io/otel/metric/instrument/syncfloat64
k8s.io/cloud-provider-vsphere/test/e2e imports
	sigs.k8s.io/cluster-api/test/framework/clusterctl imports
	sigs.k8s.io/cluster-api/controlplane/kubeadm/api/v1beta1 imports
	sigs.k8s.io/controller-runtime imports
	sigs.k8s.io/controller-runtime/pkg/manager tested by
	sigs.k8s.io/controller-runtime/pkg/manager.test imports
	sigs.k8s.io/controller-runtime/pkg/metrics/filters imports
	k8s.io/apiserver/pkg/server/options imports
	k8s.io/component-base/tracing imports
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp imports
	go.opentelemetry.io/otel/metric/instrument/syncfloat64: module go.opentelemetry.io/otel/metric@latest found (v1.16.0), but does not contain package go.opentelemetry.io/otel/metric/instrument/syncfloat64
k8s.io/cloud-provider-vsphere/test/e2e imports
	sigs.k8s.io/cluster-api/test/framework/clusterctl imports
	sigs.k8s.io/cluster-api/controlplane/kubeadm/api/v1beta1 imports
	sigs.k8s.io/controller-runtime imports
	sigs.k8s.io/controller-runtime/pkg/manager tested by
	sigs.k8s.io/controller-runtime/pkg/manager.test imports
	sigs.k8s.io/controller-runtime/pkg/metrics/filters imports
	k8s.io/apiserver/pkg/server/options imports
	k8s.io/component-base/tracing imports
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp imports
	go.opentelemetry.io/otel/metric/instrument/syncint64: module go.opentelemetry.io/otel/metric@latest found (v1.16.0), but does not contain package go.opentelemetry.io/otel/metric/instrument/syncint64

I have to manually download v0.31.0 to workaround above however I see some packages including helm.sh/helm/v3 downgraded

go get go.opentelemetry.io/otel/metric@v0.31.0
go: downgraded github.com/containerd/containerd v1.7.0 => v1.7.0-beta.3
go: downgraded go.opentelemetry.io/otel/metric v0.37.0 => v0.31.0
go: downgraded helm.sh/helm/v3 v3.12.0 => v3.12.0-dev.1
go: removed sigs.k8s.io/cluster-api/test v1.5.0

Based on above changes, I meet more compatibility issues:
(guessing helm.sh/helm/v3/pkg/kube on @v3.12.0-dev.1 error is related to k8s.io/kubectl bump to 1.28 but only guess)

go test e2e_suite_test.go
# oras.land/oras-go/pkg/auth/docker
../../../../../../pkg/mod/oras.land/oras-go@v1.2.2/pkg/auth/docker/login.go:86:39: cannot use remote (variable of type *"github.com/docker/docker/registry".Service) as "github.com/docker/docker/registry".Service value in argument to c.loginWithTLS
../../../../../../pkg/mod/oras.land/oras-go@v1.2.2/pkg/auth/docker/login_tls.go:181:30: assignment mismatch: 3 variables but registry.PingV2Registry returns 2 values
# helm.sh/helm/v3/pkg/kube
../../../../../../pkg/mod/helm.sh/helm/v3@v3.12.0-dev.1/pkg/kube/client.go:97:12: cannot use cmdutil.NewFactory(getter) (value of type "k8s.io/kubectl/pkg/cmd/util".Factory) as Factory value in struct literal: "k8s.io/kubectl/pkg/cmd/util".Factory does not implement Factory (missing method OpenAPIGetter)
FAIL	command-line-arguments [build failed]

Will capi consider to bump go.opentelemetry.io/otel/metric package in future?

I'm thinking to keep the dependencies of the e2e test in this PR as the current and try the bumps on capi after capi has newer go.opentelemetry.io/otel/metric dependencies. What do you think.

sbueringer · 2023-08-23T14:08:43Z

@wyike I opened a PR against your fork to fix it up: wyike#1
I think now it works.

As far as I can tell there are multiple paths how we depend on otel, e.g.:

go.opentelemetry.io/otel/metric
k8s.io/cloud-provider-vsphere/cmd/vsphere-cloud-controller-manager
k8s.io/cloud-provider/app
k8s.io/controller-manager/app
k8s.io/apiserver/pkg/endpoints/filters
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go.opentelemetry.io/otel/metric

 go.opentelemetry.io/otel/metric
k8s.io/cloud-provider-vsphere/test/e2e
sigs.k8s.io/cluster-api/test/framework/clusterctl
sigs.k8s.io/cluster-api/controlplane/kubeadm/api/v1beta1
sigs.k8s.io/controller-runtime
sigs.k8s.io/controller-runtime/pkg/manager
sigs.k8s.io/controller-runtime/pkg/manager.test
sigs.k8s.io/controller-runtime/pkg/metrics/filters
k8s.io/apiserver/pkg/server/options
k8s.io/apiserver/pkg/storage/storagebackend/factory
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
go.opentelemetry.io/otel/metric

Both paths are going through k8s.io/apiserver (v0.28.0). I have no idea why we somehow end up with otel versions that are higher than the ones used by k8s.io/apiserver (https://github.com/kubernetes/apiserver/blob/release-1.28/go.mod).

But it seems to help to pin otel: https://github.com/wyike/cloud-provider-vsphere/pull/1/files#r1303081043

I think neither ccm, nor CAPI, nor controller-runtime uses otel directly. It's all just through the apiserver dependency

lubronzhan · 2023-08-24T03:09:13Z

Thanks for the change
/lgtm
/approve

k8s-ci-robot · 2023-08-24T03:09:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lubronzhan, wyike

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lubronzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wyike · 2023-08-24T03:22:35Z

Since it's merged, we‘ll have a follow up PR to improve the capi bump in e2e test with Stefan's fixup wyike#1.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 23, 2023

k8s-ci-robot requested review from andrewsykim and imkin August 23, 2023 02:27

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 23, 2023

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 23, 2023

wyike force-pushed the support-1.28 branch from 6f69a12 to 7643877 Compare August 23, 2023 02:28

wyike commented Aug 23, 2023

View reviewed changes

k8s-ci-robot assigned lubronzhan and sbueringer Aug 23, 2023

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 23, 2023

wyike force-pushed the support-1.28 branch from 7643877 to 3b35b97 Compare August 23, 2023 06:43

Bump k8s to 1.28

8952cf9

wyike force-pushed the support-1.28 branch from c53dddb to 8952cf9 Compare August 23, 2023 07:40

wyike commented Aug 23, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 24, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 24, 2023

k8s-ci-robot merged commit a6fbe8f into kubernetes:master Aug 24, 2023

wyike mentioned this pull request Aug 24, 2023

Bump K8S and CAPI in e2e test #755

Merged

wyike mentioned this pull request Aug 31, 2023

REQUEST: New membership for @wyike kubernetes/org#4428

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump kubernetes 1.28 #751

Bump kubernetes 1.28 #751

wyike commented Aug 23, 2023 •

edited

Loading

k8s-ci-robot commented Aug 23, 2023

k8s-ci-robot commented Aug 23, 2023

wyike Aug 23, 2023

wyike Aug 23, 2023 •

edited

Loading

lubronzhan Aug 23, 2023

sbueringer Aug 23, 2023

wyike commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

wyike commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

wyike Aug 23, 2023 •

edited

Loading

sbueringer Aug 23, 2023

sbueringer commented Aug 23, 2023

sbueringer commented Aug 23, 2023

wyike commented Aug 23, 2023 •

edited

Loading

sbueringer commented Aug 23, 2023

lubronzhan commented Aug 24, 2023

k8s-ci-robot commented Aug 24, 2023

wyike commented Aug 24, 2023 •

edited

Loading

		@@ -166,15 +169,15 @@ require (
		k8s.io/cli-runtime v0.27.2 // indirect

Bump kubernetes 1.28 #751

Bump kubernetes 1.28 #751

Conversation

wyike commented Aug 23, 2023 • edited Loading

k8s-ci-robot commented Aug 23, 2023

k8s-ci-robot commented Aug 23, 2023

wyike Aug 23, 2023

Choose a reason for hiding this comment

wyike Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

lubronzhan Aug 23, 2023

Choose a reason for hiding this comment

sbueringer Aug 23, 2023

Choose a reason for hiding this comment

wyike commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

wyike commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

lubronzhan commented Aug 23, 2023

wyike Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

sbueringer Aug 23, 2023

Choose a reason for hiding this comment

sbueringer commented Aug 23, 2023

sbueringer commented Aug 23, 2023

wyike commented Aug 23, 2023 • edited Loading

sbueringer commented Aug 23, 2023

lubronzhan commented Aug 24, 2023

k8s-ci-robot commented Aug 24, 2023

wyike commented Aug 24, 2023 • edited Loading

wyike commented Aug 23, 2023 •

edited

Loading

wyike Aug 23, 2023 •

edited

Loading

wyike Aug 23, 2023 •

edited

Loading

wyike commented Aug 23, 2023 •

edited

Loading

wyike commented Aug 24, 2023 •

edited

Loading