Ability to choose another CNI

[okgolove]

It's rather about AWS EKS service, but it would be great to have an ability to use Calico at least.
Because I think AWS CNI pattern with private IP's and limits based on instance type breaks the main Kubernetes idea. I can't use my resources efffective.
I can't autoscale my nodes based on IP limit (cluster-autoscaler doesn't do it, kubernetes/autoscaler#1366). It seems it works as expected.
Yes, it can work for somebody. But generally it doesn't work for most cases. I have ~20 microservices with memory consuming about ~16-32mb. And I can't run them on EKS, because of IP limit.

[Bernicia]

I completely agree. My EKS cluster has to spin up more nodes even though cpu/mem are barely registering from the 10 pods on my t3.small. What a waste of resources and money, and surely not the point of EKS or AWS.

[lgg42]

I'm willing to cooperate on the scripting...but like @okgolove has said many times... VPC CNI installation seems to happens during EKS cluster creationg and not during worker node join. I've read so much today yet I can't find the part where VPC CNI just goes into the instances...

[tabern]

Ref aws/containers-roadmap#71

[AmazingTurtle]

I don't see why AWS is limiting interface ip addresses at all. Google cloud can handle this with ease, but not AWS. Why is that?

[tabern]

This has to do with the type of CNI plugin being used on the Kubernetes implementation. GKE and AWS CNI plugins are fundamentally different.

Like GKE, Amazon EKS offers a single, supported CNI plugin for managed Kubernetes clusters. On GKE, this CNI plugin is Calico, on EKS it is the AWS VPC CNI which provides native AWS VPC networking to Kubernetes clusters running on AWS.

Calico operates an overlay network for Kubernetes pods, which is fundamentally different than how AWS VPC CNI operates. This means that the Calico CNI creates a sub-network for the node and pods communicate through a single IP on the node within the cluster network space. It's sort of like an IP:port model but instead of port, the Calico CNI creates a sub IP address that it forwards traffic to.

The AWS VPC CNI does not do this, it assigns an IP to each pod that you run. In turn, the limit here is not an EKS limit, but actually the total number of IPs available in the VPC and the number of ENIs that the EC2 instance you are using can support. This approach offers several benefits including security isolation and speed. Overall however, it's not that one model is better than the other, that they just different. The AWS VPC CNI project is actively working on improving IP availability and multiple items in upcoming releases for the project intend to improve the IP availability on Kubernetes clusters using the AWS CNI.

As a side note, Calico also provides a deep set of network security and policy enforcement features. EKS does support using Calico alongside the AWS VPC CNI to provide network security and policy enforcement on EKS clusters.

[lgg42]

For anyone interested, I've written up a functional ansible role for the whole EKS bootstrapping process (including nodes management) that also supports NOT using VPC CNI but weave or other with CNI-genie. Expect bugs :-D

https://github.com/lgg42/ansible-role-eks

[jaypipes]

As @tabern mentioned, this feature request is really for EKS, and is covered in aws/containers-roadmap#71. Closing this issue because this is not actually a feature request for the VPC CNI plugin.