-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Are deployments taken care of during a kubernetes upgrade? #11371
Comments
I'm not part of the Kubespray dev team, but I've been using Kubespray on a regular basis for a number of production clusters. In my experience, you have a few options. By default, Kubespray will drain each node before upgrading it. That means eventually every Pod will be killed and recreated during the upgrade process. Kubespray will issue the equivalent of a You can disable node draining in Kubespray by adding During the upgrade, all basic components such as the kubelet, containerd, cni (ie: calico), kube-proxy... they all get restarted with their new versions. Network activity for your pods will be disrupted momentarily and this usually causes applications to crash or produce connectivity errors. You might feel that a node reboot after the upgrade could refresh and bring everything back to normal. Draining nodes is a good choice for applications that have high availability, such as microservices with multiple instances running at all times. Whem some of their pods are killed and moved during the upgrade, users won't feel any downtime. Older, monolithic, traditional applications that can only exist 1 instance at a time will suffer no matter how you chose to upgrade. They will most likely be restarted and downtime should be expected. You also have the option to run the upgrade in partial steps, ie: one node at a time. It takes a lot longer than usual (maybe days if you need to), but it will give those older applications the chance to move to other nodes in a more controlled fashion. To do this, you can run After that, you can do the same for each node simply changing the
Repeat the same process for all nodes at your convenience. The important thing to remember is to never skip minor kuberentes versions during the upgrade. For example, if you are using k8s v1.22, you should upgrade ALL NODES to v1.23, then to v1.24, then to v1.25, etc. Following that order is important because the K8s community is commited to maintaining backward compartibility between a couple of versions. If you skip too many versions, disruptions during the upgrade can get even messier. For example, the kubelet won't be able to communicate with the api-server if their versions are not compatible. For Kubespray, most of the time, that just means follow each minor Kubespray version as you upgrade, using its default supported k8s version. To be safe, whenever Kubespray has a minor release, like 2.24.0, just wait a few weeks before upgrading. There's a good chance that patch versions will follow up to fix any regressions or new bugs in the playbook (2.24.1, 2.24.2, 2.24.3 ...). Other than that, it's been working great for a few years now. |
TL;DR: |
I have a question regarding the Kubernetes upgrade process using Kubespray.
I am planning to upgrade my Kubernetes cluster with Kubespray and I am concerned about the impact on my running deployments during the upgrade.
Specifically, I would like to know:
Any insights or recommendations you can provide would be greatly appreciated.
Thanks!
The text was updated successfully, but these errors were encountered: