[EKS] [request]: Security Groups per Pod

[mike-stewart]

Tell us about your request

It would be really helpful to be able to assign Security Groups at the pod level, such that other pods on the same node don't share the same SG.

Related issue in the CNI repo: aws/amazon-vpc-cni-k8s#208

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Although Network Policies with Calico and friends can provide network segmentation within the cluster, we're missing a straightforward way to control traffic to AWS resources outside of the cluster. For example, I would like to be able to restrict access to RDS instances by referencing the pod SG in the RDS SG.

Authentication (e.g. using IAM roles) can be used in most cases to provide a layer of security, but most compliance requirements also mandate network segmentation or firewall rules as an additional defence in depth step.

Are you currently working around this issue?
My understanding is that the only workaround is to use taints/nodeAffinity to restrict pods to nodes with the right security groups, but this will result in nodes being under scheduled in a multi-tenant cluster. This is less than ideal, as it negates much of the cost benefit of moving a workload into Kubernetes.

Additional context

Attachments

[ahrkrak]

This is possible today with Tigera Secure Cloud Edition - it delivers exactly this functionality.
On Marketplace here: https://aws.amazon.com/marketplace/pp/B07FT3NTWX
Free trial link: https://tigera.io/free-ce

[phivid]

The question is: "Why paying for something that should be a native/integrated solution ?" 😊

[tabern]

We've scoped this as part of our next generation CNI plugin. Take a look and let us know what you think on #398.

Will be leaving this issue open for now as this will be a component of the CNI but not the entire project.

[dudicoco]

@tabern if this will be implemented via annotations, I suggest using the security group name/tag in the annotation rather than the sg ID to remove some hardcoded dependencies.

[minzhang28]

@mike-stewart what is the update on this ticket, does AWS has a timeline for it?

[tabern]

@minzhang28 this is part of our next generation CNI plugin. We are currently working on this.

[phivid]

@tabern, why this request has still "under consideration" label and not "proposed" or simply merged in #398 ?

[mikestef9]

Looking for some more feedback on requirements here.

One of the main issues we are addressing in the next CNI #398 is limited pod density on smaller instance types. Is there any need for security group per pod & density on the same nodes?

For reference, you can view section titled "Supported Amazon EC2 Instance Types" on this ECS doc page and look at task limit per instance to get an idea of how many pods per instance will be supported if you choose the security group per pod networking mode.

[hendrikhalkow]

I think security groups per pod should be available regardless of the node size as I’d like to avoid redesigning my workload because I changed the infrastructure. Not having this on small nodes would also cause additional headaches when spot instances are used because you need to exclude these small machines when using security groups per pod.

Higher pod density is always nice but for me not as important as the first thing. @mikestef9 your comment sounds like these two things might be mutually exclusive?

Can’t you increase the IP address limit per ENI?

[yann-soubeyrand]

In my opinion, the more pod density on a node you have, the more diverse the workloads you'll have on each node and so the more needed the security group per pod will be. But I guess this will not always be the case.

[bhks]

I think security groups per pod should be available regardless of the node size as I’d like to avoid redesigning my workload because I changed the infrastructure. Not having this on small nodes would also cause additional headaches when spot instances are used because you need to exclude these small machines when using security groups per pod.

I am not an expert, but could network policies be a better approach for inside cluster communication security. From / to outside definitely security groups or some others.
Also would be happy to understand more on the use case where we need per pod security groups.

[dsilvasc]

Simple use case: I'd like to deploy two services to the same node, and only have one of them allowed to talk to DAX, or ElasticSearch, or ElastiCache.

[mike-stewart]

@bhagwat070919 Kubernetes network policies are great for managing traffic between Kubernetes resources, but being able to assign Security Groups to pods would address a major gap in EKS network security. Namely, securing traffic between pods and AWS resources like RDS, ElastiCache, etc.

These resources don't have a static IP, so Network Policies are no help here unless you're using a provider like Cillium that can do layer 7 egress policies. Even if you were to segregate access to AWS resources using egress Network policies at Layer 7, it still requires that you provide a way too permissive ingress policies on the RDS/ElastiCache Security Group - not ideal from a security perspective.

We need to be able to give RDS a Security Group, which allows traffic from a Security Group that is assigned to a Kubernetes pod. This needs to be done at the granularity of the pod rather than the node, to enforce least-privilege permissions and allow all sorts of multi-tenant use-cases.

@mikestef9 Regarding whether pod density and SGs per pod are both needed at once, it would certainly be easier if the two weren't mutually exclusive. That said, I see the network security issue as being more urgent & important for the workloads I've got as it's a compliance blocker, whereas the pod density problem can be worked around in a lot of cases.

[bhks]

@dsilvasc,

Simple use case: I'd like to deploy two services to the same node, and only have one of them allowed to talk to DAX, or ElasticSearch, or ElastiCache.

Given its an outbound connection and security from pod to elasticsearch or rds which can be done via private link and IAM for service accounts. Where IAM for service accounts helps to allow only pods which admin wants to assign the IAM role not to the node.

Just a thought on the design for now:
Given Kubernetes system does a pretty job at scheduling pods based on node labels. I would separate the critical pods to a groups of nodes with specific security groups to which I can schedule all critical pods apps which needs access to lets say Elastic search or RDS but I would still give access to pods with IAM for service account and have the VPC linked to private link.

[eriksw]

Scenario: Multiple databases or other protected resources that live in a VPC but outside kubernetes with access controlled by security groups, where each pod/service should have access to a different subset of those external-to-k8s databases.

In my view the ask for per-pod security groups is so that we can run pods on the same nodes that have access to (and/or are able to be accessed by) different sets of things on the network, where those different things are defined on the basis of security group rules. (A vpcSecurityGroupSelector usable in k8s network policies for both ingress and egress might meet my needs, but I have a feeling it'd be a bit of a minefield compared to security-group-per-pod.)

[ranmeekkhsc]

are we looking at any timelines to close this ?

[sftim]

This might be done: see https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html

[mikestef9]

Security groups per pod is now available!

For more details:
What's New
Launch Blog
Documentation

[mikestef9]

Hey one thing to add here, we are aware of an issue where pods running on branch network interfaces are failing TCP readiness and liveness checks. This will result in pods failing to start. To workaround this issue, you can change the readiness checks to use commands or UDP. We are working on a fix to be included in an upcoming release of the VPC CNI plugin.

Edit 9/16 - Fix has been released on GitHub.

[sc-juho]

The web page https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html mentions that:

Traffic flow to and from pods with associated security groups are not subjected to Calico network policy enforcement and are limited to Amazon EC2 security group enforcement only. Community effort is underway to remove this limitation.

Could somebody link to a relevant github issue or similar where this community effort is tracked?

[mikestef9]

https://github.com/projectcalico/felix/issues/2425