chore(networking): bump aws-vpc-cni version to 1.15.5

[moshevayner]

What this PR does / why we need it:
Bump aws cni to version 1.15.5.
Once 1.29 release branch is created, I'll bump master to 1.16.0 which was released a couple of days ago, to allow an additional buffer between the versions, if that makes sense.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Special notes for your reviewer:
https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.15.5

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johngmyers for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[moshevayner]

/retest

[moshevayner]

/retest

[moshevayner]

/retest

[hakman]

Please ignore pull-kops-e2e-aws-upgrade-k127-ko127-to-klatest-kolatest-many-addons.
/test pull-kops-e2e-cni-amazonvpc
/lgtm

[moshevayner]

@hakman I can't seem to figure out from the logs why is the amazonvpc job failing.
Trying to spin up a cluster on my personal account using a build from this branch to see if it comes up.
Converting to draft for the time being.

[moshevayner]

So, I have a feeling something with the newly introduced network policy support is causing a cascading failure that begins with (probably) coredns failing:

 k -n kube-system logs coredns-867c995cd5-d9gxs
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = 9617ae84604c33431b2bf5a8d8e93450b34eb11d1103af1b1962d4d016b8eb111bde503da621c2bf37a233cfe70c25b727b74f78133b9bcf4b6191e897f96fa8
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:44830->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:33291->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:43286->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:41628->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:55739->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:48107->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:55021->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:37836->172.20.0.2:53: i/o timeout
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://100.64.0.1:443/version": dial tcp 100.64.0.1:443: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:52611->172.20.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 7001883994134103369.583922312319829419. HINFO: read udp 172.20.66.221:59047->172.20.0.2:53: i/o timeout
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://100.64.0.1:443/version": dial tcp 100.64.0.1:443: i/o timeout
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://100.64.0.1:443/version": dial tcp 100.64.0.1:443: i/o timeout
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s

Seems like it can't access the k8s api, and from there it all goes south.

It's weird, because according to the docs, the enforcements of netpols is disabled by default, but there might be something else that is eluding me at this point.

I'll have to keep digging and understand what's the root cause.
Keeping this here for extra transparency.

[moshevayner]

@hakman any chance this is related? aws/amazon-vpc-cni-k8s#2103
I just found it there. It seems to be over a year old, but unless I'm mistaken, we are now using Ubuntu 2204 by default, and coredns uses the kubernetes ClusterIP service to interact with the API.
WDYT?
cc @olemarkus

[hakman]

/test pull-kops-e2e-cni-amazonvpc

[hakman]

@hakman any chance this is related? aws/amazon-vpc-cni-k8s#2103 I just found it there. It seems to be over a year old, but unless I'm mistaken, we are now using Ubuntu 2204 by default, and coredns uses the kubernetes ClusterIP service to interact with the API. WDYT?

Due to that issue, we run AWS VPC CNI tests with Ubuntu 20.04. From what I see, failures look like flakes in the tests, but I may be wrong.

[hakman]

/test pull-kops-e2e-cni-amazonvpc

[hakman]

/test pull-kops-e2e-cni-amazonvpc

[k8s-ci-robot]

@moshevayner: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kops-e2e-aws-upgrade-k127-ko127-to-klatest-kolatest-many-addons	eda667f	link	false	/test pull-kops-e2e-aws-upgrade-k127-ko127-to-klatest-kolatest-many-addons
pull-kops-e2e-cni-amazonvpc	eda667f	link	false	/test pull-kops-e2e-cni-amazonvpc

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[moshevayner]

@hakman any chance this is related? aws/amazon-vpc-cni-k8s#2103 I just found it there. It seems to be over a year old, but unless I'm mistaken, we are now using Ubuntu 2204 by default, and coredns uses the kubernetes ClusterIP service to interact with the API. WDYT?

Due to that issue, we run AWS VPC CNI tests with Ubuntu 20.04. From what I see, failures look like flakes in the tests, but I may be wrong.

I think these are legit errors, based on this comment (just making sure you saw it, since it might've been lost in the pile of comments here).

I created a cluster locally using a binary built from this branch, and got the mentioned results.

[hakman]

@hakman any chance this is related? aws/amazon-vpc-cni-k8s#2103 I just found it there. It seems to be over a year old, but unless I'm mistaken, we are now using Ubuntu 2204 by default, and coredns uses the kubernetes ClusterIP service to interact with the API. WDYT?

Due to that issue, we run AWS VPC CNI tests with Ubuntu 20.04. From what I see, failures look like flakes in the tests, but I may be wrong.

I think these are legit errors, based on this comment (just making sure you saw it, since it might've been lost in the pile of comments here).

I created a cluster locally using a binary built from this branch, and got the mentioned results.

I think you are right 😄...

[Deshke]

via https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh#L104

  # Temporary fix for https://github.com/aws/amazon-vpc-cni-k8s/pull/2118
  sudo sed -i "s/^MACAddressPolicy=.*/MACAddressPolicy=none/" /usr/lib/systemd/network/99-default.link || true

how could this be added?

[hakman]

via https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh#L104

  # Temporary fix for https://github.com/aws/amazon-vpc-cni-k8s/pull/2118
  sudo sed -i "s/^MACAddressPolicy=.*/MACAddressPolicy=none/" /usr/lib/systemd/network/99-default.link || true

how could this be added?

Please create a separate issue for this.

[hakman]

@moshevayner could we try updating to 1.16.2?

[moshevayner]

@moshevayner could we try updating to 1.16.2?

Sure, I'll give that a try asap and update!

[moshevayner]

This PR is preceded by #16297

/close

[k8s-ci-robot]

@moshevayner: Closed this PR.

In response to this:

This PR is preceded by #16297

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.