CNI does not wait for ENI and IP to be available before reporting ready when setting MINIMUM_IP_TARGET and WARM_IP_TARGET #1724
Comments
|
@dtnyn - What is your WARM_IP_TARGET and MINIMUM_IP_TARGET values? Do you have custom networking enabled? With respect to WARM_IP_TARGET/MINIMUM_IP_TARGET and WARM_ENI_TARGET, there is no difference on how IPAMD starts up. We do wait for ipamd to start - https://github.com/aws/amazon-vpc-cni-k8s/blob/release-1.9/scripts/entrypoint.sh#L139 before copying cni config file - https://github.com/aws/amazon-vpc-cni-k8s/blob/release-1.9/scripts/entrypoint.sh#L145 and in ipamd init - we do try to assign IPs to the primary ENI - https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.9.0/pkg/ipamd/ipamd.go#L493. So before node ready typically we should have primary ENI with ips in non-custom networking mode. Once you share the configs, we can try to verify this scenario locally. |
|
These are the config enabled for the CNI This is used on both |
|
Thanks for sharing the details. Since you mentioned this is a new node, the instance will have only the primary ENI attached and during aws-node initialization IPs won't be assigned to the primary ENI since custom networking is enabled. The secondary ENI will be allocated by the reconciler based on the WARM targets requirement since available IPs will be 0. Before the reconciler thread starts node can be marked as ready and datastore will not have any IPs. But we will try both the configs locally and get back to you. |
|
Maybe a better description of the problem would be the difference between the init time and the time when the datastore is first populated with. We're seeing an average wait of up to 3 minutes when During that 3 waits minutes pods are stuck pending for an extended period of time since there's no IP to be allocated to them. I do sometimes see pods waiting on the default configuration with Examples:
|
|
@dtnyn Have you given the new IP prefix's a try/do they have similar delay? I expect I might be experiencing similar delays as we have the same type of config in our clusters. |
|
I did a test on my local cluster to compare the time difference between init time and when datastore has IPs and didn't find any difference with respect to the configs - WARM_ENI_TARGET = 1 Init start -> Reconciler increasing the pool and added a secondary ENI -> WARM_IP_TARGET = 1 and MINIMUM_IP_TARGET = 3 Init start -> Reconciler increasing the pool and added a secondary ENI -> Both the cases within 4-5 seconds reconciler tried to increase the pool. There is no difference in the code path for both the cases. Once the reconciler starts based on the WARM targets pool will be increased. But I feel there is one optimization that can be done here, since you are using custom networking and primary ENI cannot have IPs/Prefixes attached we should go ahead and attach secondary ENI based on the WARM targets and that should help with the pod launch time otherwise we will have to wait for the reconciler to increase the pool and node will be marked as ready. |
|
@sidewinder12s We have tried prefix mode but did not do extended testing due to the minimum allocation size of @jayanthvn I believe that your proposed optimization, attaching a new ENI with pre-allocated IP, will best for our current problem. Since we can observe that the ENI attachment and the time starting the reconcile are very fast. Only the process of the initial reconiller adding new IP to the ENI that we see is very slow. Any subsequent reconciller processes on the the same ENI is fast. |
|
@dtnyn - Yes that's right. I have the PR and testing it out locally. |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
@jayanthvn we're experiencing this issue as well. How has your local testing been going? |
|
I am waiting for the code review. |
|
+1 |
|
Any updates on PR Progress? |
|
We had discussed few changes for the PR but couldn't progress because of other commitments. I will take it up next week for changes/review and track it for 1.11.5 or 1.12.0 release. Will provide the release timelines once the PR is merged. |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
Not stale. We still regularly see this issue. |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
Not stale |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days |
|
Not stale. |
|
I faced the same issue while creating EKS cluster with custom networking with prefix delegation and I got below error - {"level":"error","ts":"2023-03-21T16:32:26.571Z","caller":"ipamd/ipamd.go:845","msg":"Failed to increase pool size due to not able to allocate ENI AllocENI:error attaching ENI: attachENI: failed to attach ENI: InvalidParameterCombination: You may not attach a network interface to an instance if they are not in the same availability zone\n\tstatus code: 400, request id: 8160c309-fa43-4b0f-95f2-45b3b5100ed7"} {"level":"debug","ts":"2023-03-21T17:26:17.714Z","caller":"datastore/data_store.go:646","msg":"AssignIPv4Address: IP address pool stats: total: 0, assigned 0"} {"level":"debug","ts":"2023-03-21T17:26:17.714Z","caller":"datastore/data_store.go:646","msg":"AssignPodIPv4Address: ENI eni-0b881d8fd9bae3c92 does not haveavailable addresses"} I followed docs here and here and I have available Ip's in the subnets and the parameters WARM_IP_TARGET and AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG are set properly. when I have /16 subnets , it works fine it seems and when I have primary node subnets with /24 and secondary ICDR subnets with /18 , then I get this issue. |
|
Closing as fixed by #2354. This will ship in the next VPC CNI release |
|
What happened:
After setting
MINIMUM_IP_TARGETandWARM_IP_TARGET. When scheduled on a new node, pods are stuck onfailed to assign an IP address to containerfor a few minutes before being able to be created.From my observation I can see that after setting these override, aws-node does not wait for its datastore to be populated first before reporting ready causing pods to try to be scheduled onto these nodes and goes into a crashing state as no IP can be assigned
Pod log
I have been monitoring the ready state for the pod and see that it's declared ready before ipamd is able to successfully add the ENI and its IP to the datastore
The pod status change
Readiness
The node status change
Inside ipamd around that time when the pod and node are declared ready I can see pods are trying to be scheduled and assigned an IP but there's nothing in the datastore at the time
It is not until a while later that ipamd successfully populate the datastore
During this time period between
aws-nodereporting as ready and when ipamd datastore is actually populated with IPs, pods are being scheduled on and failing which is undesirable.What you expected to happen:
I expected a similar behaviour observed with the default
WARM_ENI_TARGET=1behaviour where the pod is not considered to be ready until the datastore adds the attached ENI is populated with IPs.The following logs are from using the default option with no
MINIMUM_IP_TARGETandWARM_IP_TARGEToverride. In this case aws-cni works as expectedPod status
Ready status
I can see in the ipamd log that the datastore successfully populate itself with the ENI's IP around this time
Environment:
kubectl version):v1.20.11v1.9.0cat /etc/os-release):Ubuntu 20.04.3 LTS (Focal Fossa)uname -a):Linux xxxxxxxx 5.11.0-1019-aws #20~20.04.1-Ubuntu SMP Tue Sep 21 10:40:39 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: