Error: "Failed to generate 10-awsconflist" error="json: cannot unmarshal number into Go struct field NetConf.plugins.mtu of type string"

[tomsucho]

What happened:

On EKS 1.23 updated AWS VPC CNI addon to 1.12.2-eksbuild.1 as recommended in AWS docs

Attach logs

the log from the aws-node pod says:

$ kubectl -n kube-system logs aws-node-jtlzf
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-v4-cni
time="2023-02-28T13:56:09Z" level=info msg="Starting IPAM daemon... "
time="2023-02-28T13:56:09Z" level=info msg="Checking for IPAM connectivity... "
time="2023-02-28T13:56:10Z" level=info msg="Copying config file... "
time="2023-02-28T13:56:10Z" level=error msg="Failed to generate 10-awsconflist" error="json: cannot unmarshal number into Go struct field NetConf.plugins.mtu of type string"

What you expected to happen:
The aws-node pod to come up fine.

How to reproduce it (as minimally and precisely as possible):
Tried to update AWS VPC CNI addon to latest as per the AWS docs.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.16-eks-48e63af", GitCommit:"e6332a8a3feb9e0fe3db851878f88cb73d49dd7a", GitTreeState:"clean", BuildDate:"2023-01-24T19:18:15Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• CNI Version
  from: amazon-k8s-cni:v1.11.4-eksbuild.1 to: amazon-k8s-cni:v1.12.2-eksbuild.1
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):

UPDATE: this only happens when this is set in cfg:

        - name: ENABLE_BANDWIDTH_PLUGIN
          value: "true"

[jdn5126]

Hi @tomsucho, sorry for the delay. I was able to reproduce this issue, and the failure happens here: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/cmd/aws-vpc-cni/main.go#L258

When ENABLE_BANDWIDTH_PLUGIN is set to "true", we call json.Unmarshal from golang, and that fails parsing the MTU set for the egress-v4-cni plugin: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/10-aws.conflist#L18

That MTU needs to be a string, i.e. 9001 -> "9001". I will expedite a fix for this, but for existing releases, if you want to enable the bandwidth plugin, I think the best way is to not set the environment variable and manually modify /etc/cni/net.d/10-aws.conflist on each node to add the following entry (at the bottom):

{
  "type": "bandwidth",
  "capabilities": {
    "bandwidth": true
  }
}

[jdn5126]

This will ship in the next 1.12 release, i.e. v1.12.6

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.