Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add toleration with toleration seconds condition #24041

Closed
wants to merge 1 commit into from
Closed

Add toleration with toleration seconds condition #24041

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,14 @@ means that if this pod is running and a matching taint is added to the node, the
the pod will stay bound to the node for 3600 seconds, and then be evicted. If the
taint is removed before that time, the pod will not be evicted.

{{< note >}}
The evicted pod with `tolerationSeconds` field has the possibility to be scheduled to the prior node again. To avoid such behavior:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are lacking some subsections for the whole "concepts" section. @sftim, could you advice?

In particular, I feel like there should be subtitle before line 132.

Other than that, we have to stress the fact that, even with "tolerationSeconds", this is still a toleration, and for that reason the Pod can get scheduled in the same node again.

Instead of "to avoid such behavior", I would say "If this is not what you want", then "you can add a NoSchedule taint additionally to the NoExecute taint".

Then, I don't think you have to distinguish system-level taints (what is that anyways?). Only the second point is relevant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, feel free to add a heading before line 132 - no objection from me for doing that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that then that heading seems isolated. We don't have a heading for "taints" or "tolerations". But maybe we can fix that in a follow up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think system-level taints is like node.kubernetes.io/unreachable when node is down. And controller will add both NoExcute and NoSchedule taint to the node.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's already covered in https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-nodes-by-condition

The fact that they are added automatically doesn't change how you write tolerations for them :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, PR #25998 is improving some of the documentation in this area.

- If the node is added system-level taints, Kubernetes will be responsible for applying `NoExecute` taint, as well as `Noschedule` taint.
So the pod won't be scheduled to the prior node.
Comment on lines +153 to +154
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhouya0 is this another way of writing what you meant?

Suggested change
- If the node is added system-level taints, Kubernetes will be responsible for applying `NoExecute` taint, as well as `Noschedule` taint.
So the pod won't be scheduled to the prior node.
- If the node is tainted based on kubelet or node status , Kubernetes is responsible for applying the `NoExecute` and `NoSchedule` taints. These taints prevent the scheduler placing the replacement Pod onto the same node.

- If the node is tainted by system admin, the best practice is to apply both `NoExecute` and `NoSchedule` taints; otherwise, the pod may be
scheduled and evicted back and forth.
Comment on lines +155 to +156
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- If the node is tainted by system admin, the best practice is to apply both `NoExecute` and `NoSchedule` taints; otherwise, the pod may be
scheduled and evicted back and forth.
- If you are setting a `NoExecute` taint manually on a node, you should normally also
set `NoSchedule`. Otherwise, any new Pod that could run on this node may be scheduled
onto, it but then immediately face eviction because of the `NoExecute` taint.

{{< /note >}}

## Example Use Cases

Taints and tolerations are a flexible way to steer pods *away* from nodes or evict
Expand Down