Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncordon nodes that failed draining during update #6545

Closed
thegreenbear opened this issue Aug 17, 2020 · 2 comments
Closed

Uncordon nodes that failed draining during update #6545

thegreenbear opened this issue Aug 17, 2020 · 2 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@thegreenbear
Copy link
Contributor

What would you like to be added:
When running the upgrade-cluster.yml playbook, I would like nodes that were cordoned but failed to be drained to be uncordoned.

Why is this needed:
During the upgrade of the cluster, Kubespray cordons and then drain nodes before actually updating them.
However, if a node fails to be drained (for example because of no poddisruption budget), Kubespray will leave it cordoned.
This means repeated failing runs (CI/CD) would cause more and more nodes to be left cordoned creating capacity issues on the running cluster.

The enhanced behavior I propose is the following: if the node fails to drain, no upgrade task was actually performed yet.
So it is safe to uncordon the node. However the upgrade should STILL fail as the upgrade did not work.

@thegreenbear thegreenbear added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 17, 2020
@floryut
Copy link
Member

floryut commented Aug 27, 2020

/close

(merged)

@k8s-ci-robot
Copy link
Contributor

@floryut: Closing this issue.

In response to this:

/close

(merged)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

3 participants