Compatibility with K8s 1.29

[tnqn]

Create the issue to track compatibility with impending K8s 1.29.

Changes that may matter now

• Kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate is now beta and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a kubernetes.io/legacy-token-invalid-since label if the credentials have not been used in the time specified by --legacy-service-account-token-clean-up-period (defaulting to one year), and are referenced from the .secrets list of a ServiceAccount object, and are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by --legacy-service-account-token-clean-up-period (defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the kubernetes.io/legacy-token-invalid-since label. (LegacyServiceAccountTokenCleanUp beta kubernetes/kubernetes#120682)

  • It may invalidate the manually created service account token antctl-service-account-token and antctl-service-account-token if they are classified as legacy auto-generated. If that's the case, we should consider stopping create such tokens and update docs.
  • @luolanzone is checking if this would affect the two tokens.

• Allow specifying ExternalTrafficPolicy for Services with ExternalIPs. (Allow specifying ExternalTrafficPolicy for Services with ExternalIPs kubernetes/kubernetes#119150)

  • Antrea proxyAll mode doesn't hornor ExternalTrafficPolicy when Service type is ClusterIP, will be fixed by Support Local ExternalTrafficPolicy for Services with ExternalIPs #5795

Changes that may matter in the future

• Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses (KEP-1880 Multiple ServiceCIDR ( and IPAddress allocation) kubernetes/kubernetes#116516)
  • The API is in alpha. When it becomes mature, Service ClusterIPs could be allocated from different subnets, Antrea should change how it discovers the ServiceCIDRs accordingly.
• Kube-proxy now has a new nftables-based mode (kube-proxy nftables backend kubernetes/kubernetes#121046)
  • The mode is in alpha. When it becomes mature, Antrea should be able to run with kube-proxy of this mode.
• Added a new ipMode field to the .status of Services where type is set to LoadBalancer. The new field is behind the LoadBalancerIPMode feature gate. (Make Kubernetes aware of the LoadBalancer behaviour kubernetes/kubernetes#119937)
  • Already tracked by Implement KEP-1860 in AntreaProxy: Make Kubernetes aware of the LoadBalancer behavior #5342 (the version info about 1.28 is wrong)

Tests

• Conformance test K8s v1.29.0-rc.0 + Antrea v1.14.1 on Linux
• Conformance test K8s v1.29.0 + Antrea v1.14.1 on Linux (https://github.com/antrea-io/antrea/actions/runs/7217146969)
• sig-network test K8s v1.29.0 + Antrea v1.14.1 on Linux (https://github.com/antrea-io/antrea/actions/runs/7217153283)
• Conformance test K8s v1.29.0 + Antrea v1.14.1 on Windows

Items

• Bump k8s.io Go dependencies to 0.29 Upgrade k8s libraries to v0.29.2 #5843 by @hjiajing

[luolanzone]

Regarding the feature gate LegacyServiceAccountTokenCleanUp, I have went through the codes and verified in the Kind cluster with v1.29.0 images. A token will be treated as an auto-generated secret-based toke only when it's in the .secrets list of a ServiceAccount object. Our service account tokens antctl-service-account-token and antrea-agent-service-account-token are created manually by the manifest antrea.yaml and doesn't show up in the .secrets list of SA antctl and antrea-agent. These two tokens might be marked with a label kubernetes.io/legacy-token-last-used if they are used at least once. But they won't be cleaned up automatically by the legacy token cleaner controller since they are not referenced from the .secrets list of a ServiceAccount object

The conclusion is that the LegacyServiceAccountTokenCleanUp feature gate should have no impact to Antrea even it's enabled by default in v1.29.0.

[antoninbas]

Not completely related to this issue, but it may be nice to update our k8s.io Go dependencies to 0.29. I think it's been a while since we bumped them up.

[tnqn]

Not completely related to this issue, but it may be nice to update our k8s.io Go dependencies to 0.29. I think it's been a while since we bumped them up.

Sure, it's also a necessary step to access the new ServiceCIDR API and the ipMode field. Added it as one item of the issue.

[XinShuYang]

Windows conformance test passed on 1.29 cluster.
windows_conformance_result_129.txt

[hjiajing]

Maybe I can take the taske Bump k8s.io Go dependencies to 0.29.

[luolanzone]

@hjiajing Thanks for taking the sub-task, you can refer to this PR #4935 as a reference.

[prakrit55]

hey @luolanzone, @antoninbas can I try to bump dependencies to 0.29, if @hjiajing is occupied

[antoninbas]

hey @luolanzone, @antoninbas can I try to bump dependencies to 0.29, if @hjiajing is occupied

@hjiajing is still actively working on this in #5843

[antoninbas]

All tasks completed