Allow reconciler retry for InsufficientCIDR EC2 error

[jayanthvn]

What type of PR is this?
enhancement

Which issue does this PR fix:
N/A

What does this PR do / Why do we need it:

1. Without custom networking, nodeInit tries to attach secondary IPs or prefixes (https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/ipamd/ipamd.go#L493) and if this fails because of InsufficientCIDR then aws-node restarts which will increase the number of DescribeNetworkInterface calls hence instead allow the reconciler to retry.
2. With custom networking, nodeInit will skip attaching secondary IPs or prefixes to the primary ENI. Reconciler will try to increase the pool (https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/ipamd/ipamd.go#L691) but primary ENI will be skipped and reconciler will try to allocate a new ENI and attach IPs/prefixes which might fail because of InsufficientCIDR, then reconciler tries to create a new ENI instead allow the reconciler to retry on the previously created ENI.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
n/a

Testing done on this change:

{"level":"warn","ts":"2021-08-23T02:10:11.695Z","caller":"ipamd/ipamd.go:825","msg":"failed to allocate all available IPv4 Prefixes on ENI eni-0ebee6fb5784f13bf, err: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 65f5461d-46f0-46ac-9efc-7f7d3c1b0fff"}
{"level":"error","ts":"2021-08-23T02:10:11.992Z","caller":"ipamd/ipamd.go:825","msg":"failed to allocate one IPv4 prefix on ENI eni-0ebee6fb5784f13bf, err: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 08bc8ef1-c737-4345-9ab4-fa8b8d3bac1f"}
{"level":"error","ts":"2021-08-23T02:10:11.992Z","caller":"ipamd/ipamd.go:579","msg":"InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 08bc8ef1-c737-4345-9ab4-fa8b8d3bac1f"}
{"level":"error","ts":"2021-08-23T02:10:11.992Z","caller":"ipamd/ipamd.go:579","msg":"Unable to attach IP/Prefixes for the ENI, reconciler will try again"}

After first try, reconciler will wait for 2 mins before next retry -

grep InsufficientCidr /var/log/aws-routed-eni/ipamd.log
{"level":"error","ts":"2021-08-26T22:28:28.035Z","caller":"ipamd/ipamd.go:791","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-005389d15c5d28157: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 527f91e7-a3de-4709-be32-2af26cdd8c23"}
{"level":"warn","ts":"2021-08-26T22:28:28.035Z","caller":"ipamd/ipamd.go:741","msg":"Failed to allocate 1 IP addresses on an ENI: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 527f91e7-a3de-4709-be32-2af26cdd8c23"}
{"level":"debug","ts":"2021-08-26T22:28:33.227Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:28:38.227Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:28:43.227Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:28:48.228Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:28:53.228Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:28:58.228Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:03.229Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:08.229Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:13.229Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:18.230Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:23.230Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:28.230Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:33.240Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:38.240Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:43.240Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:48.241Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:53.241Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:29:58.242Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:30:03.242Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:30:08.243Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:30:13.243Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:30:18.243Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"debug","ts":"2021-08-26T22:30:23.244Z","caller":"ipamd/ipamd.go:590","msg":"Recently we had InsufficientCidr Error hence will wait for 2 mins before retrying"}
{"level":"error","ts":"2021-08-26T22:30:28.597Z","caller":"ipamd/ipamd.go:895","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-005389d15c5d28157: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: b80aca0f-2457-4942-94e1-1d60bd8ec3bc"}
{"level":"warn","ts":"2021-08-26T22:30:28.597Z","caller":"ipamd/ipamd.go:840","msg":"failed to allocate all available IPv4 Prefixes on ENI eni-005389d15c5d28157, err: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: b80aca0f-2457-4942-94e1-1d60bd8ec3bc"}
{"level":"error","ts":"2021-08-26T22:30:28.941Z","caller":"ipamd/ipamd.go:899","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-005389d15c5d28157: InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request.\n\tstatus code: 400, request id: 30543ae4-5f1f-42a6-8c52-ea2e359d89b5"}

Automation added to e2e:

No

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No

Does this change require updates to the CNI daemonset config files to work?:

No

Does this PR introduce any user-facing change?:

No

Once InsufficientCidrError is identified, IPAMD will wait for 2 mins to retry. This is done to reduce the number AllocIPAddresses call. During this time even if a CIDR frees up, IPAMD will wait to come out of the cool down.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[M00nF1sh]

/lgtm