Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kubeadm upgrade docs #4770

Merged
merged 9 commits into from
Sep 16, 2017
47 changes: 23 additions & 24 deletions _data/tasks.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
bigheader: "Tasks"
abstract: "Step-by-step instructions for performing operations with Kubernetes."
bigheader: "Tasks"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert the addition of the extra spacing.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry, didn't notice this. It's due to an Atom package that automatically formats YAML files.

abstract: "Step-by-step instructions for performing operations with Kubernetes."
toc:
- docs/tasks/index.md

- title: Install Tools
- title: Install Tools
section:
- docs/tasks/tools/install-kubectl.md
- docs/tasks/tools/install-minikube.md
- docs/setup/independent/install-kubeadm.md

- title: Configure Pods and Containers
- title: Configure Pods and Containers
section:
- docs/tasks/configure-pod-container/assign-cpu-ram-container.md
- docs/tasks/configure-pod-container/configure-volume-storage.md
Expand All @@ -28,7 +28,7 @@ toc:
- docs/tasks/configure-pod-container/configure-pod-configmap.md
- docs/tools/kompose/user-guide.md

- title: Inject Data Into Applications
- title: Inject Data Into Applications
section:
- docs/tasks/inject-data-application/define-command-argument-container.md
- docs/tasks/inject-data-application/define-environment-variable-container.md
Expand All @@ -37,7 +37,7 @@ toc:
- docs/tasks/inject-data-application/distribute-credentials-secure.md
- docs/tasks/inject-data-application/podpreset.md

- title: Run Applications
- title: Run Applications
section:
- docs/tasks/run-application/run-stateless-application-deployment.md
- docs/tasks/run-application/run-single-instance-stateful-application.md
Expand All @@ -51,13 +51,13 @@ toc:
- docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md
- docs/tasks/run-application/configure-pdb.md

- title: Run Jobs
- title: Run Jobs
section:
- docs/tasks/job/parallel-processing-expansion.md
- docs/tasks/job/coarse-parallel-processing-work-queue/index.md
- docs/tasks/job/fine-parallel-processing-work-queue/index.md

- title: Access Applications in a Cluster
- title: Access Applications in a Cluster
section:
- docs/tasks/access-application-cluster/web-ui-dashboard.md
- docs/tasks/access-application-cluster/access-cluster.md
Expand All @@ -70,10 +70,10 @@ toc:
- docs/tasks/access-application-cluster/configure-cloud-provider-firewall.md
- docs/tasks/access-application-cluster/list-all-running-container-images.md
- docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume.md
- title: Configuring DNS for a Cluster
path: https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/cluster-dns
- title: Configuring DNS for a Cluster
path: https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/cluster-dns

- title: Monitor, Log, and Debug
- title: Monitor, Log, and Debug
section:
- docs/tasks/debug-application-cluster/resource-usage-monitoring.md
- docs/tasks/debug-application-cluster/get-shell-running-container.md
Expand All @@ -90,10 +90,10 @@ toc:
- docs/tasks/debug-application-cluster/debug-stateful-set.md
- docs/tasks/debug-application-cluster/debug-application-introspection.md
- docs/tasks/debug-application-cluster/audit.md
- title: Use Explorer to Examine the Runtime Environment
path: https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/explorer
- title: Use Explorer to Examine the Runtime Environment
path: https://github.com/kubernetes/kubernetes/tree/release-1.5/examples/explorer

- title: Access and Extend the Kubernetes API
- title: Access and Extend the Kubernetes API
section:
- docs/tasks/access-kubernetes-api/http-proxy-access-api.md
- docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions.md
Expand All @@ -102,11 +102,11 @@ toc:
- docs/tasks/access-kubernetes-api/configure-aggregation-layer.md
- docs/tasks/access-kubernetes-api/setup-extension-api-server.md

- title: TLS
- title: TLS
section:
- docs/tasks/tls/managing-tls-in-a-cluster.md

- title: Administer a Cluster
- title: Administer a Cluster
section:
- docs/tasks/administer-cluster/access-cluster-api.md
- docs/tasks/administer-cluster/access-cluster-services.md
Expand All @@ -122,12 +122,13 @@ toc:
- docs/tasks/administer-cluster/cluster-management.md
- docs/tasks/administer-cluster/upgrade-1-6.md
- docs/tasks/administer-cluster/kubeadm-upgrade-1-7.md
- docs/tasks/administer-cluster/kubeadm-upgrade-cmd.md
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kubeadm-upgrade-1.8 please?

- docs/tasks/administer-cluster/namespaces.md
- docs/tasks/administer-cluster/namespaces-walkthrough.md
- docs/tasks/administer-cluster/dns-horizontal-autoscaling.md
- docs/tasks/administer-cluster/safely-drain-node.md
- docs/tasks/administer-cluster/declare-network-policy.md
- title: Install Network Policy Provider
- title: Install Network Policy Provider
section:
- docs/tasks/administer-cluster/calico-network-policy.md
- docs/tasks/administer-cluster/romana-network-policy.md
Expand All @@ -142,10 +143,10 @@ toc:
- docs/tasks/administer-cluster/configure-multiple-schedulers.md
- docs/tasks/administer-cluster/ip-masq-agent.md
- docs/tasks/administer-cluster/dns-custom-nameservers.md
- title: Change Cluster Size
path: https://github.com/kubernetes/kubernetes/wiki/User-FAQ#how-do-i-change-the-size-of-my-cluster/
- title: Change Cluster Size
path: https://github.com/kubernetes/kubernetes/wiki/User-FAQ#how-do-i-change-the-size-of-my-cluster/

- title: Federation - Run an App on Multiple Clusters
- title: Federation - Run an App on Multiple Clusters
section:
- docs/tasks/federation/federation-service-discovery.md
- docs/tasks/federation/set-up-cluster-federation-kubefed.md
Expand All @@ -161,13 +162,11 @@ toc:
- docs/tasks/administer-federation/replicaset.md
- docs/tasks/administer-federation/secret.md

- title: Manage Cluster Daemons
- title: Manage Cluster Daemons
section:
- docs/tasks/manage-daemon/update-daemon-set.md
- docs/tasks/manage-daemon/rollback-daemon-set.md

- title: Manage GPUs
- title: Manage GPUs
section:
- docs/tasks/manage-gpus/scheduling-gpus.md


54 changes: 54 additions & 0 deletions docs/tasks/administer-cluster/kubeadm-upgrade-cmd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
approvers:
- pipejakob
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and luxas ;)
and @roberthbailey and @jbeda I guess...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

he he, a copy and paste thing, yes makes sense …

title: Upgrading kubeadm clusters
---

{% capture overview %}

This guide is for upgrading kubeadm clusters from version 1.7.x to 1.8.x.
See also [upgrading kubeadm clusters from 1.6 to 1.7](/docs/tasks/administer-cluster/kubeadm-upgrade-1-7/) if you're on a 1.6 cluster, currently.

{% endcapture %}

{% capture prerequisites %}
You need to have a Kubernetes cluster running version 1.7.x in order to use the process described here. Note that only one minor version upgrade is supported, that is, you can only upgrade from, say 1.7 to 1.8, not from 1.7 to 1.9.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: v1.7.0 or higher

Maybe we can move the note down to the out-of-scope section?


Before proceeding, make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v180-alpha2) carefully.

The following is out of scope for `kubeadm upgrade`, that is, you need to take care of it yourself:

- No etcd upgrades are performed. You can, for example, use `etcdctl` to take care of this.
- Any app-level state, for example, a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we break this out to a section where you're saying the general: "As a best-practice; you should back up what's important to you" or something like that
We should clarify that this should not touch any workloads, just the k8s internal components, but it's always good to be on the safe side



Note that `kubeadm upgrade` is 'eventually idempotent', that is, you can run it over and over again if you find yourself in a bad state and it should be able to recover.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this down to the recovery section?


{% endcapture %}

{% capture steps %}

## On the master
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we name this Upgrading your control plane?

Then remind the user that this has to be done on the master


1. Upgrade `kubectl` using [curl](/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl). Note: DO NOT use `apt` or `yum` or any other package manager to upgrade it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only upgrading kubeadm manually is actually required.
kubelet will be upgraded later using the debs/rpms

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, now I'm a bit confused. At least that was the state when we discussed it. Has this changed or did I misunderstand you here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unnecessary. Only kubeadm is needed


2. Install the most recent version of `kubeadm` using curl.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please tell us how to do that:

$ export VERSION=v1.8.0 # or any given released k8s version
$ export ARCH=amd64 # or arm, arm64, ppc64le or s390x
$ curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm


3. On the master node, run `kubeadm upgrade plan`, which tells you what versions are available.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add these in code blocks:
```
$ kubeadm upgrade plan
```

and paste some preliminary output (I have that on the upgrades PR).

Please explain what this command does as well, like:

  • It checks that the cluster is in an upgradeable state
  • It fetches the versions available to upgrade to in an user-friendly way


4. Pick a version to upgrade to and run, for example, `kubeadm upgrade apply --version v1.7.3`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this in a code block as well so it's more discoverable; and with some simple outputs.
Please tell the user what's happening:

  • It checks that the cluster is in an upgradeable state (API Server reachable, all nodes in the Ready state, control plane healthy)
  • It enforces the version skew policies
  • It makes sure the control plane images are available or available to pull to the machine
  • It upgrades the control plane components or rollbacks if any of them fails to come up
  • It applies the new kube-dns and kube-proxy manifests and enforces all necessary RBAC rules are created


5. After `kubeadm upgrade` you need to manually upgrade your SDN.

Your CNI provider might have its own upgrade instructions to follow now.
Check the [addons](/docs/concepts/cluster-administration/addons/) page to
find your CNI provider and see if there are additional upgrade steps
necessary.

## Recovering from a bad state

You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: If kubeadm upgrade somehow fails and fails to rollback (due to an unexpected shutdown during execution for instance) you may run kubeadm upgrade again, as it is idempotent and should eventually make sure the actual state is the desired state that you are declaring.


{% endcapture %}

{% include templates/task.md %}