This repository has been archived by the owner on Sep 4, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 466
kube-aws: Drain nodes before shutting them down #465
Closed
Closed
Commits on Aug 8, 2016
-
Configuration menu - View commit details
-
Copy full SHA for 2f547b6 - Browse repository at this point
Copy the full SHA 2f547b6View commit details -
Configuration menu - View commit details
-
Copy full SHA for d4bd382 - Browse repository at this point
Copy the full SHA d4bd382View commit details -
Configuration menu - View commit details
-
Copy full SHA for dde38f6 - Browse repository at this point
Copy the full SHA dde38f6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 81a2729 - Browse repository at this point
Copy the full SHA 81a2729View commit details -
Configuration menu - View commit details
-
Copy full SHA for 56a77fe - Browse repository at this point
Copy the full SHA 56a77feView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5296214 - Browse repository at this point
Copy the full SHA 5296214View commit details -
Configuration menu - View commit details
-
Copy full SHA for a84a6f4 - Browse repository at this point
Copy the full SHA a84a6f4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 65bd4fe - Browse repository at this point
Copy the full SHA 65bd4feView commit details
Commits on Aug 9, 2016
-
aws: cluster upgrade support via
kube-aws up --update
Does rolling replacement update on controller ASG, followed by workers Punts on upgrading etcd cluster- simply makes sure resource definitions don't change after create.
Configuration menu - View commit details
-
Copy full SHA for a3c772a - Browse repository at this point
Copy the full SHA a3c772aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3dff28f - Browse repository at this point
Copy the full SHA 3dff28fView commit details
Commits on Aug 11, 2016
-
kube-aws: Drain nodes before shutting them down to give running pods …
…time to gracefully stop. This change basically achieves it by running `docker run IMAGE kubectl drain THE_NODE --force` on the to-be-shut-down node before the kubelet gets SIGTERM in a CoreOS' shutdown process. Without this change, kubelets getting SIGTERM without a prior `drain` results in unfunctional(actually not schedulable) pods to have statuses `Ready`. With this change, when an ASG's desired cap decreased, a node's status changes over time as follows: On desired cap change: STATUS=Ready On shut-down started: STATUS=Ready,SchedulingDisabled (<- Pods are stopped and status is changed by `kubectl drain`) On shut-down finished: Status=NotReady,SchedulingDisabled (<- It's `NotReady` but it won't result in a down-time because we already stopped both further scheduling and pods) After a minute: The node disappears from the output of `kubectl get nodes` Note that: * This applies to manual shutdowns(via running `sudo systemctl shutdown` for example) and automated shutdowns(triggered by AWS AutoScaling when nodes get rotated out of a group) * We currently depend on the community docker image `mumoshu/kubectl` because `kubectl` included in the official `coreos/hyperkube` image doesn't work due to the issue kubernetes/kubernetes#24088 in Kubernetes. Once the issue is fixed and the CoreOS team published the new hyperkube image with the updated Kubernetes, we can remove that dependency. * The author considers this an experimental feature. So you shouldn't expect configuration API regarding this stable. It may change in the future. Re-format code introduced in the previous with gofmt to conform the rules and make the build pases
Configuration menu - View commit details
-
Copy full SHA for 24c650b - Browse repository at this point
Copy the full SHA 24c650bView commit details -
fix kube-node-drainer not to fail when draining a node running Daemon…
…Set-managed pods Before this change, you could reproduce the failure running `sudo sytemctl stop kube-node-drainer` which showed a error message indicating this issue like: ``` May 09 05:04:08 ip-10-0-0-234.ap-northeast-1.compute.internal sh[15376]: error: DaemonSet-managed pods: cassandra-e3tdb (use --ignore-daemonsets to ignore) May 09 05:04:08 ip-10-0-0-234.ap-northeast-1.compute.internal systemd[1]: kube-node-drainer.service: Control process exited, code=exited status=1 ``` After this change, you can drain a node even if it was running DaemonSet-managed pods: ``` May 09 10:41:08 ip-10-0-0-202.ap-northeast-1.compute.internal sh[7671]: node "ip-10-0-0-202.ap-northeast-1.compute.internal" cordoned May 09 10:41:08 ip-10-0-0-202.ap-northeast-1.compute.internal sh[7671]: WARNING: Skipping DaemonSet-managed pods: cassandra-jzfqo May 09 10:41:08 ip-10-0-0-202.ap-northeast-1.compute.internal sh[7671]: node "ip-10-0-0-202.ap-northeast-1.compute.internal" drained ```
Configuration menu - View commit details
-
Copy full SHA for d2a9085 - Browse repository at this point
Copy the full SHA d2a9085View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f90853 - Browse repository at this point
Copy the full SHA 8f90853View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.