External network connectivity issue with EKS CNI #1966
Assignees
Comments
|
@bpradipt @EmmEff is it possible to collect some diagnostic data on EKS? Create a regular (runc) pod and execute the following commands in the pod with According to the documentation, the EKS CNI plugin explicitly sets a static ARP entry. If so, I think we can fix the issue by setting the same ARP entry in a network namespace in a peer pod VM. |
|
@yoheiueda please find the requested details O/p from regular runc pod O/p from the worker node |
|
@bpradipt Thank you very much! The output of The output of The output of An ARP entry for this MAC address is explicitly set in the pod network as follows. So, I think we can fix the connectivity issue by setting this ARP entry like this @bpradipt could you create a peer pod and try this work around to check whether the external connectivity issue is fixed or not? You can identify a MAC address as described above. |
|
Another thing I noticed is that jumbo frames (MTU 9001) are enabled on EKS. The current implementation of peer pods restricts a maximum MTU size to be not greater than 1450. (#68) I am not sure this will cause connectivity issue or not. I think TCP connections are not affected since MSS is negosiated during TCP handshakes. UDP packets initiated from a peer pod will not be affected, since a smaller MTU size is used. UDP traffics initiated from a regular pod to a peer pod will be fragmented. If path MTU Discovery does not work due to peer pods, large packets will be dropped. Anyway, jumbo frames should be supported with peer pods from the performance perspective, so I will investigate how we can adjust MTU. |
|
Awesome @yoheiueda. I tried your suggestion and it fixes the issue :-) |
yoheiueda
added a commit
to yoheiueda/cloud-api-adaptor
that referenced
this issue
Aug 7, 2024
Permanent ARP entries that are explicitly set in the network namespace for a pod in a worker node are propagated to the podns network namespace in a peer pod VM. The EKS CNI plugin sets such a permanent ARP entry, and this patch is necessary for peer pods to work with EKS networks. Fixes confidential-containers#1966 Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
This was referenced Aug 7, 2024
Merged
yoheiueda
added a commit
to yoheiueda/cloud-api-adaptor
that referenced
this issue
Aug 16, 2024
Permanent ARP entries that are explicitly set in the network namespace for a pod in a worker node are propagated to the podns network namespace in a peer pod VM. The EKS CNI plugin sets such a permanent ARP entry, and this patch is necessary for peer pods to work with EKS networks. Fixes confidential-containers#1966 Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
yoheiueda
added a commit
to yoheiueda/cloud-api-adaptor
that referenced
this issue
Aug 16, 2024
Permanent ARP entries that are explicitly set in the network namespace for a pod in a worker node are propagated to the podns network namespace in a peer pod VM. The EKS CNI plugin sets such a permanent ARP entry, and this patch is necessary for peer pods to work with EKS networks. Fixes confidential-containers#1966 Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
bpradipt
pushed a commit
that referenced
this issue
Aug 29, 2024
Permanent ARP entries that are explicitly set in the network namespace for a pod in a worker node are propagated to the podns network namespace in a peer pod VM. The EKS CNI plugin sets such a permanent ARP entry, and this patch is necessary for peer pods to work with EKS networks. Fixes #1966 Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As reported at #1920 (comment), peer pod network has external network connectivity issue with EKS CNI.
The design of the CNI plugin for Kubernetes networking over AWS VPC is described here.
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/cni-proposal.md#solution-components
The text was updated successfully, but these errors were encountered: