Upgrade vpc cni 1.28

[jaskaransarkaria]

Tests:

• addon successfully upgrades ✅
• Int tests are passing ✅
• enis are being span up correctly and pods are getting ips assigned correctly (tested up to 5000 pods) ✅
• nothing unusual in aws-node logs ✅
• ipamd node logs checked on new nodes, ips are successfully assigned ✅
• cluster upgraded to 1.28 and pods scaled to 800 successfully (bringing up new nodes) ✅

Subnet discovery is enabled by default. VPC-CNI will pick the subnet with the most number of free IPs from the nodes' VPC/AZ to create the secondary ENIs. The subnets considered are the subnet the node is created in and subnets tagged with `kubernetes.io/role/cni`.
If `ENABLE_SUBNET_DISCOVERY` is set to `false` or if DescribeSubnets fails due to IAM permissions, all secondary ENIs will be created in the subnet the node is created in.

ENABLE_SUBNET_DISCOVERY --> We should set this to false to preserve our current behaviour. Even if this is set to true the behaviour should be the same as we don't use the kubernetes.io/role/cni tag on our subnets. Setting this to false protects us from future module changes which may bring in this tagging.

https://github.com/aws/amazon-vpc-cni-k8s/blob/8f9253e2e4452fe0e9e6a26a05675c8b7ae7a8fe/README.md?plain=1#L548

---------------------------------------------------------------------------------------------------------
      aws-node-5d2wk-before.yaml                              |       aws-node-6k6jv-after.yaml
---------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------

  creationTimestamp: "2024-06-06T08:36:42Z"                   |   annotations:
                                                              >     github_teams: all-org-members
                                                              >   creationTimestamp: "2024-06-06T11:04:44Z"
    controller-revision-hash: 956849f86                       |     controller-revision-hash: b6d79ffcf
    pod-template-generation: "2"                              |     pod-template-generation: "4"
  name: aws-node-5d2wk                                        |   name: aws-node-6k6jv
  resourceVersion: "1261"                                     |   resourceVersion: "60129"
  uid: 58be49eb-fcef-407e-8449-dda7239385a7                   |   uid: ad0192c8-7847-4805-a4e3-6bcf87331875
            - ip-172-20-100-78.eu-west-2.compute.internal     |             - ip-172-20-66-221.eu-west-2.compute.internal
    - name: CLUSTER_ENDPOINT                                  <
      value: https://128E90F5A4726655B0A3E60C704382FF.gr7.eu- <
                                                              >     - name: ENABLE_SUBNET_DISCOVERY
                                                              >       value: "false"
      value: v1.17.1                                          |       value: v1.18.1
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo
      name: kube-api-access-skkxm                             |       name: kube-api-access-8pjn8
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo
      name: kube-api-access-skkxm                             |       name: kube-api-access-8pjn8
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo
      name: kube-api-access-skkxm                             |       name: kube-api-access-8pjn8
  nodeName: ip-172-20-100-78.eu-west-2.compute.internal       |   nodeName: ip-172-20-66-221.eu-west-2.compute.internal
  - name: kube-api-access-skkxm                               |   - name: kube-api-access-8pjn8
    lastTransitionTime: "2024-06-06T08:36:52Z"                |     lastTransitionTime: "2024-06-06T11:04:46Z"
    lastTransitionTime: "2024-06-06T08:36:55Z"                |     lastTransitionTime: "2024-06-06T11:04:48Z"
    lastTransitionTime: "2024-06-06T08:36:55Z"                |     lastTransitionTime: "2024-06-06T11:04:48Z"
    lastTransitionTime: "2024-06-06T08:36:42Z"                |     lastTransitionTime: "2024-06-06T11:04:44Z"
  - containerID: containerd://f42de97d4400152829d1c18252bcf35 |   - containerID: containerd://38bf0bc4d35383055d9be7e24494f62
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo
    imageID: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/ama |     imageID: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/ama
        startedAt: "2024-06-06T08:36:55Z"                     |         startedAt: "2024-06-06T11:04:46Z"
  - containerID: containerd://74f46d6b455631eece98a676775695b |   - containerID: containerd://5890d26e56a03e087400b3f6304cf0c
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 066635153087.dkr.ecr.il-central-1.amazonaws.com/am
    imageID: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/ama |     imageID: sha256:86800e25303d102ce2d081833ccb7b51b354d610f
        startedAt: "2024-06-06T08:36:54Z"                     |         startedAt: "2024-06-06T11:04:46Z"
  hostIP: 172.20.100.78                                       |   hostIP: 172.20.66.221
  - containerID: containerd://ab6f04b3b77788583f7233a5f8aeb43 |   - containerID: containerd://6a2e93eec3af91af2094197de59f205
    image: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazo |     image: 066635153087.dkr.ecr.il-central-1.amazonaws.com/am
    imageID: 602401143452.dkr.ecr.eu-west-2.amazonaws.com/ama |     imageID: sha256:3f0789d4b13f6149abed0acb024fc383f0569f98e
        containerID: containerd://ab6f04b3b77788583f7233a5f8a |         containerID: containerd://6a2e93eec3af91af2094197de59
        finishedAt: "2024-06-06T08:36:47Z"                    |         finishedAt: "2024-06-06T11:04:45Z"
        startedAt: "2024-06-06T08:36:47Z"                     |         startedAt: "2024-06-06T11:04:45Z"
  podIP: 172.20.100.78                                        |   podIP: 172.20.66.221
  - ip: 172.20.100.78                                         |   - ip: 172.20.66.221
  startTime: "2024-06-06T08:36:42Z"                           |   startTime: "2024-06-06T11:04:44Z"
---------------------------------------------------------------------------------------------------------

Additional reading:

https://aws.amazon.com/blogs/containers/amazon-vpc-cni-introduces-enhanced-subnet-discovery/