[EKS] [Fargate] [request]: fargate daemon set support

[runningman84]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Allow us to run daemon sets like kube2iam, fluentbit and others in a fargate env.

What do you want us to build?
Improve the scheduler to allow running whitelisted daemon sets on fargate nodes.

Which service(s) is this request for?
Fargate, EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Right it is very difficult to run fargate as a replacement for ec2 nodes because you do not have exporter logs, iam integration and so on...

Are you currently working around this issue?
Use fargate only for non ciritical tasks which do not need iam, logs or anything else...

[mikestef9]

Hey @runningman84,

You can track #701 for a logging solution for EKS/Fargate.

Can you explain more on the IAM integration point and why you are using kube2iam? IAM roles for service accounts works today with EKS/Fargate.

[runningman84]

Yes IAM stuff can be solved using the native integration. But there is still software which runs an old sdk.

I think there should be a general solution for daemon sets in fargate.

[dalgibbard]

FWIW; we would like daemon-sets in EKS Fargate for two main reasons:

• Parity between mixed workloads on EC2 + Fargate (daemonsets that can perform the same actions irrelevant of underlying node type, for consistancy/compliance)
• To offer k8s as a 'platform as a service' to our internal users: specifically, allowing them to deploy whatever they need/want, without needing to worry about things like datadog sidecars (just for example) in their configurations.

This has the potential added advantage that the Datadog Daemonset in particular supports log exporting too (as well as container metrics and events etc), so removes our current blocker of log centralisation for compliance as an extra benefit.

We looked at https://github.com/tumblr/k8s-sidecar-injector for example to auto-inject sidecars, but really, this is exactly the purpose of a daemonset for our use case, and sidecars have limited scope compared to daemonsets.

[mziyabo]

Similar to @dalgibbard I ended up writing a mutating webhook that injects sidecars on fargate specifically - https://github.com/mziyabo/fargate-eks-sidecar-injector.
Could give someone else an idea on how to tackle this while we wait.

[t0mbombadil]

Lack of this ability actually cripples AWS GuardDuty EKS Runtime Monitoring

When this feature is enabled and you are using Fargate, GuardDuty shows that your EKS
clusters are covered by Runtime Monitoring and healthy and even it deploys automatically guardduty-agent daemonset
to your EKS clusters, but if you inspect it closely you will see that it's not doing anything on Fargate nodes.

So basically it's even worse if GuardDuty would say to you that your EKS clusters are not covered in RuntimeMonitoring, because it gives you false belief that they are

kubectl get daemonset aws-guardduty-agent  -n amazon-guardduty -o yaml  
...
 spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
              - key: eks.amazonaws.com/compute-type
                operator: NotIn
                values:
                - fargate
...

Also beside GuardDuty runtime monitoring, it also stops us from using 3rd party tools to achieve this same, that run as a daemonset, for example - https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html