diff --git a/keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md b/keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md new file mode 100644 index 00000000000..c05cb783f0f --- /dev/null +++ b/keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md @@ -0,0 +1,365 @@ +--- +title: Advanced configurations with kubeadm (Kustomize) +authors: + - "@fabriziopandini" +owning-sig: sig-cluster-lifecycle +participating-sigs: + - sig-cluster-lifecycle +reviewers: + - "@neolit123" + - "@rosti" + - "@ereslibre" + - "@detiber" + - "@vincepri" + - "@chuckha" +approvers: + - "@timothysc" + - "@luxas" +editor: "@fabriziopandini" +creation-date: 2019-07-22 +last-updated: 2019-07-22 +status: implementable +--- + +# Advanced configurations with kubeadm (Kustomize) + +## Table of Contents + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Story 3](#story-3) + - [Implementation Details](#implementation-details) + - [Kustomize integration with kubeadm](#kustomize-integration-with-kubeadm) + - [Providing and storing Kustomize patches to kubeadm](#providing-and-storing-kustomize-patches-to-kubeadm) + - [Storing and retrieving Kustomize patches for kubeadm](#storing-and-retrieving-kustomize-patches-for-kubeadm) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + + +## Release Signoff Checklist + +- [x] kubernetes/enhancements issue in release milestone, which links to KEP +- [x] KEP approvers have set the KEP status to `implementable` +- [x] Design details are appropriately documented +- [x] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [ ] Graduation criteria is in place +- [x] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + +## Summary + +This KEP is aimed at defining a new kubeadm feature that will allow users to bootstrap +a Kubernetes cluster with static pods customizations not supported by the Kubeadm component configuration. + +## Motivation + +Kubeadm currently allows you to define a limited set of configuration options for a +Kubernetes cluster via the Kubeadm component configuration or the corresponding CLI flags. + +More specifically the kubeadm component configuration provides an abstraction that allows to: + +1. To define configurations settings at cluster level using the `ClusterConfiguration` + config object +2. To define a limited set of configurations at the node level using the + `NodeRegistrationOptions` object or the `localAPIEndpoint` object + +This abstraction is well suited for the most common cluster configurations/use cases, but +there are other use cases that cannot be achieved with the kubeadm component configuration +as of today. Some examples: + +- It is not possible to add sidecars e.g. for authorization web-hooks serving components. +- It is not possible to set/alter timeouts for liveness probes in control plane components. + +This KEP aims to introduce a new feature that will eneable users full control of static +pod manifest generated by Kubeadm at node level - vs the kubeadm component configuration +that allows mostly cluster-wide configurations on control-plane/etcd args only -. + +With this new feature users should not be required anymore to manually alter static +pod manifests stored into `/etc/kubernetes/manifests` after kubeadm init/join. + +### Goals + +Considering the complexity of this topic, this document is expected to be subject +to some iterations. The goal of the current iteration is to: + +- Get initial approval on Summary and Motivation paragraphs +- To identify a semantic for defining “advanced configurations” for control-plane/etcd + static pod manifests. +- To define UX for passing “advanced configurations” to kubeadm init and to kubeadm join. +- To define mechanics, limitations, and constraints for preserving “advanced configurations” + during cluster lifecycle and more specifically for supporting the kubeadm upgrade workflow. +- To ensure the proper functioning of “advanced configurations” with kubeadm phases. +- To clarify what is in the scope of kubeadm and what instead should be the responsibility + of the users/of higher-level tools in the stack like e.g. cluster API + +### Non-Goals + +- To provide any validation or guarantee about the security, conformance, + consistency, of “advanced configurations” for control-plane/etcd settings. + As it is already for `extraArgs` fields in the kubeadm component configuration or in the + Kubelet/KubeProxy component config, the responsibility of proper usage of those + advanced configuration options belongs to higher-level tools/users. +- To deprecate the Kubeadm component configuration because: + - The component configuration provides an abstraction well suited for most common use + cases (that can be addressed with cluster-wide configurations on control-plane/etcd command line args only) + - The component configuration implicitly defines the main cluster variants the kubeadm team + is committed to support and monitor in the Kubernetes test grid. +- To define how to manage “advanced configurations” when kubeadm and [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm) + project will integrate. This will be defined in following iterations of this KEP. +- To define how to manage “advanced configurations” for the addons + (this is postponed until kubeadm - [`addons`](https://github.com/kubernetes-sigs/addon-operators) + project integration). + +## Proposal + +### User Stories + +#### Story 1 +As a cluster administrator, I want to add a sidecar to the kube-apiserver pod for running an +authorization web-hooks serving component. + +#### Story 2 +As a cluster administrator, I want to set timeouts for the kube-apiserver liveness +probes for edge clusters. + +#### Story 3 +As a cluster administrator, I want to upgrade my cluster preserving all the +“advanced configuration” already in place + +### Implementation Details + +This proposal explores as a first option for implementing Kubeadm +“advanced configurations“ the usage of Kustomize; please refer to +[Declarative application management in Kubernetes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md) +and [Kustomize KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/0008-kustomize.md) +for background information about Kustomize. + +#### Kustomize integration with kubeadm + +By adopting Kustomize in the context of Kubeadm, this proposal assumes to: + +- Let kubeadm generate the static pod manifests as usual. +- Use kubeadm generated artifacts as a starting point for applying patches + containing “advanced configurations”. + +This has some implications: + +1. The higher-level tools/users have to express “advanced configurations” using + one of the two alternative techniques supported by Kustomize - the [strategic + merge patch](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#patchstrategicmerge) + and the [JSON patch](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#patchjson6902) -. +2. The higher-level tools/users have to provide patches before running kubeadm; + this point is going to be further discussed in the following paragraphs. +3. Kubeadm is responsible for coordinating the execution of Kustomize within the + init/join/upgrade workflows +4. as a consequence of the previous point, higher-level tools/users are not + requested to take care of defining `kustomization.yaml` files nor to define + a local folder structure. +5. `patchesJson6902` can't be used without a `kustomization.yaml` file defining `target` + for a patch; this limitation is not considered blocking for starting implementation; + however a solution for this problem should be defined before graduating to beta + (e.g. require `kustomization.yaml` in case the users want to use `patchesJson6902`). + +Additionally, in order to simplify the first implementation of this KEP, this +proposal is going to assume that Kustomize patches for kubeadm are always defined +specifically for the node where kubeadm is going to be executed. + +This point could be reconsidered in the future, by e.g. introducing cluster-wide +patches and/or patches for a subset of nodes. + +The resulting workflow for using in kustomize will be the following + +1. Create a folder with some patches, e.g. + +``` +mkdir kubeadm-patches + +cat <./kubeadm-patches/patch1.yaml +apiVersion: v1 +kind: Pod +metadata: + name: kube-apiserver + namespace: kube-system +{your kube-apiserver patch here} +EOF + +cat <./kubeadm-patches/patch2.yaml +apiVersion: v1 +kind: Pod +metadata: + name: kube-controller-manager + namespace: kube-system +{your kube-controller-manager patch here} +EOF +``` + +2. Run kubeadm passing the new patch, e.g. + +``` +kubeadm init --experimental-kustomize kubeadm-patches/ +or +kubeadm init phase control-plane ... --experimental-kustomize kubeadm-patches/ +``` +or +``` +kubeadm join --experimental-kustomize kubeadm-patches/ +or +kubeadm join phase control-plane-prepare control-plane --experimental-kustomize kubeadm-patches/ +``` +or +``` +kubeadm upgrade apply --experimental-kustomize kubeadm-patches/ +``` +or +``` +kubeadm upgrade node --experimental-kustomize kubeadm-patches/ +or +kubeadm upgrade node phase control-plane --experimental-kustomize kubeadm-patches/ +``` + +NB1. `--experimental-kustomize` is the proposed name for the flag to be renamed to --kustomize` +when beta is reached. `-k` abbreviation can be reserved or even fully connected. + +#### Providing and storing Kustomize patches to kubeadm + +Before kubeadm init, Kustomize patches should be eventually provided to kubeadm +by higher-level tools/users; patches should be defined in a custom location +on the machine file system and this location could be passed to +kubeadm init/join with a CLI flag. + +In order to simplify the first implementation of this KEP, this proposal is assuming +to use the same approach also for kubeadm join; this point could be reconsidered +in the future, by e.g. defining a method for allowing higher-level tools/users to +define Kustomize patches using a new CRD. + +#### Storing and retrieving Kustomize patches for kubeadm + +Kustomize patches, should be preserved during the whole cluster lifecycle, mainly +for allowing kubeadm to preserve changes during the kubeadm upgrade workflow. + +In order to simplify the first implementation of this KEP, this proposal is assuming +that Kustomize patches will remain stored in the custom location on the machine file system +for the necessary time, and that this location will be passed to kubeadm upgrade with a CLI +flag; this point could be reconsidered in the future, by e.g. defining a method for +allowing higher-level tools/users to define Kustomize patches using a new CRD. + +### Risks and Mitigations + +_Confusion between kubeadm component configuration and kustomize_ +Kubeadm already offers a way to implement cluster settings, that is the kubeadm component +configuration and component configs. Adding a new feature for supporting “advanced configurations” +can create confusion in the users. + +kubeadm maintainers should take care of making differences cristal clear in release notes +and feature announcement: + +- The component configuration provides an abstraction well suited for most common use cases + that can be addressed with cluster-wide configurations on control-plane/etcd command line args only, + while “advanced configurations”/kustomize allows users full control of static + pod manifest generated by Kubeadm at node level. +- The component configuration implicitly defines the main cluster variants the kubeadm team is + committed to support and monitor in the Kubernetes test grid, while instead higher-level + tools/user are responsible for the security, conformance, consistency, of + “advanced configurations” for control-plane/etcd static pod manifests. + +_Misleading expectations on the level of flexibility_ +In order to provide guarantee about kubeadm respecting “advanced configurations” during +init, join, upgrades or single-phase execution, it is necessary to define some trade-offs +around _what_ can be customized and _how_. + +Even if the proposed solution is based on the user feedback/issues, the kubeadm maintainers +want to be sure the implementation is providing the expected level of flexibility and, in +order to ensure that, we will wait for at least one K8s release cycle for the users to provide +feedback before moving forward in graduating the feature to beta. + +Similarly, the kubeadm maintainers should work with [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm) +project and [`addons`](https://github.com/kubernetes-sigs/addon-operators) project +to ensure a consistent approach across different components. + +_Component version change during upgrades_ +Static pod manifests are managed by kubeadm, while “advanced configurations”/kustomize patches +will be managed by users. + +It might happen that a static pod manifest during upgrades is changed by kubeadm in a way that +some patches will not cleanly apply anymore. + +The kubeadm maintainers will work on release notes to make potential breaking changes more visible; +additionally, upgrade instructions will be updated adding the recommendation to --dry-run and check +expected changes before upgrades. + +_Kustomize errors during kubeadm workflows_ +When executing “advanced configurations”/kustomize patches within kubeadm workflows, we are introducing +an external element that can potentially generate errors during commands execution. + +Error management should be adapted to this new possible risk, ensuring that the node remains in a +consistent state in case of errors. + +## Design Details + +### Test Plan + +Add at least one periodic E2E test job exercising “advanced configurations” during init, +join and upgrades. + +Please note that, in accordance with the split of responsibilities defined in the previous paragraphs, +the new E2E test will focus _only_ on the mechanics of applying “advanced configurations”/kustomize +patches, not on the possible combination of patches and nor on the security, conformance, consistency, +of the resulting Kubernetes cluster. + +### Graduation Criteria + +This proposal in its initial version covers only the creation of a new alpha feature. +Graduation criteria will be defined in the following iterations on this proposal and +consider user feedback as well. + +### Upgrade / Downgrade Strategy + +As stated in goals, kubeadm will preserve “advanced configurations” during upgrades, +and more specifically, it will re-apply patches after each upgrade. + +Downgrades are not supported by kubeadm, and not considered by this proposal. + +### Version Skew Strategy + +This proposal does not impact kubeadm compliance with official K8s version skew policy; +higher-level tools/user are responsible for the security, conformance, consistency of +“advanced configurations” that can impact or the aforementioned point. + +## Implementation History + +Tuesday, July 30, 2019 +- the `Summary` and `Motivation` sections being merged signaling SIG acceptance +- the `Proposal` section being merged signaling agreement on a proposed design +- the date implementation started + +## Drawbacks + +Kubeadm already offers a way to implement cluster settings, that is the kubeadm API +and the support for providing component configs. Adding a new feature for supporting +“advanced configurations” can create confusion in the users. + +See risks and mitigations. + +## Alternatives + +There are many alternatives to “Kustomize” in the ecosystem; see [Declarative application management in Kubernetes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md). + +While there is great value in several different approaches “Kustomize” was selected as +the first choice for this proposal because it already has first-class supported in +kubectl (starting from v1.14).