Behavior change in daemonset pod eviction

[jdn5126]

Which component are you using?:
cluster-autoscaler

What version of the component are you using?:
v1.21.3

Component version: v1.21.3

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.7-eks-4721010", GitCommit:"b77d9473a02fbfa834afa67d677fd12d690b195f", GitTreeState:"clean", BuildDate:"2022-06-27T22:22:16Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.13-eks-15b7512", GitCommit:"94138dfbea757d7aaf3b205419578ef186dd5efb", GitTreeState:"clean", BuildDate:"2022-08-31T19:15:48Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?:
AWS EKS

What did you expect to happen?:
Daemonset pods, especially those marked system-node-critical, to either not be evicted by default or to be evicted only after all other pods are evicted.

What happened instead?:
Daemonset pods were evicted before other non-critical and non-daemonset pods.

How to reproduce it (as minimally and precisely as possible):

1. Deploy daemonset pod without the cluster-autoscaler.kubernetes.io/enable-ds-eviction annotation and without --daemonset-eviction-for-occupied-nodes=false.
2. Deploy other pods on node with high termination grace period
3. Schedule node eviction
4. See that daemonset pod is evicted before other pods

Anything else we need to know?:
I understand that cluster-autoscaler.kubernetes.io/enable-ds-eviction and --daemonset-eviction-for-occupied-nodes were added to control daemonset pod eviction, but the default behavior on occupied nodes changed from "do not evict daemonset pods" to "evict daemonset pods unless annotation is present". This behavior change can lead to regressions and unexpected changes.

If the default behavior cannot be changed back, it seems like daemonset pods, or even more specifically system-node-critical pods, should not be evicted until all other pods are evicted. Adding issue #4337 for referencing similar discussion

[jdn5126]

Any thoughts on this?

[Kuzbekov]

We faced the same situation in GKE (1.24.6-gke.1500) with managed local dns service. Our services have slight delay while stopping, but local dns usually stops faster than some workloads and that leads to failed requests.
The problem is that we can't change needed settings of autoscaler, and can't change annotations of daemonset because it could be overwritten by cloud.

[dims]

xref: kubernetes-sigs/aws-fsx-csi-driver#253

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.