tracking issue for self-hosting solutions and proposals

[neolit123]

tracking issue for self-hosting solutions and proposals.

related to:
#1072

the SelfHosting alpha feature gate is being deprecated in 1.12 and removed in 1.13 for a couple of main reasons:

• the feature can be qualified as difficult to maintain in it's current form
• no feedback or support from the community for this alpha feature

in more than one SIG Cluster Lifecycle meetings it was discussed that instead of tight coupling this into the kubeadm bootstrap process, a separate command / tooling is the better approach.

ideally, this feature would need feedback and contributions by the community and it's not a high priority for SIG Cluster Lifecycle and the kubeadm maintainers.

[stealthybox]

Discussed during kubeadm office hours - Wed, Aug 22

Evaluate in the future whether to move alpha phases for self-hosting features into a separate sub-command (where the the user explicitly opts-in to the feature)

Should consider implementing un-pivoting -- right now, there is no supported downgrade path for SelfHosted enabled clusters

Thanks @mikebryant for much-needed user feedback

[ereslibre]

ideally, this feature would need feedback and contributions by the community and it's not a high priority for SIG Cluster Lifecycle and the kubeadm maintainers.

At SUSE we are interested in this feature for the kubic project, but I think it would be good to understand the full context of why this feature is being deprecated and removed (to understand the root causes of the complexities that it introduces), because we are starting to use kubeadm as the main way of deploying and mantaining a Kubernetes cluster. We don't really depend on this feature, as it's a PoC that we are using SelfHosting at this stage.

From what was discussed on the last meetings, it seems that the direction is to start a KEP with a design proposal and work in that route, right?

Given our interest in the SelfHosting feature (whether it's in the current shape or something different in the future, started out of the KEP to be proposed), we are interested in the development of this feature and in general in the design of it.

[neolit123]

From what was discussed on the last meetings, it seems that the direction is to start a KEP with a design proposal and work in that route, right?

yes, before starting the work on the KEP it might be a good idea to book kubeadm office hours time for this as an agenda item, and gather some more feedback.

ideally somewhere in the next k8s cycle.

edit: we also use google docs a lot of ideas and WIP proposals.
you can start working on the KEP in a gdoc form and share the document for an extended read.

Given our interest in the SelfHosting feature (whether it's in the current shape or something different in the future, started out of the KEP to be proposed), we are interested in the development of this feature and in general in the design of it.

sounds good!

[neolit123]

closed by:
kubernetes/kubernetes#69878

[MarkusTeufelberger]

What's the current direction on self-hosted Kubernetes? There's https://github.com/kubernetes-incubator/bootkube in incubation for quite a while now and according to kubernetes-retired/bootkube#97 they initially wanted to add this functionality to kubeadm. Is this now reverted and their tool is the one to be used for self-hosting exclusively, is self-hosting no longer a goal for Kubernetes in general or will kubeadm still strive eventually to install clusters self-hosted (but the way to get there isn't clear)?

[neolit123]

in 1.13 we moved self-hosting outside of the standard init workflow and into a kubeadm alpha sub-command.
you can still "pivot" a cluster to use self-hosting from kubeadm but upgrades are unsupported.

from my perspective the future of this feature is unclear in terms of k8s as a whole, but for kubeadm it was decided that we want to have it optional and mostly driven by community contributions.

[MarkusTeufelberger]

Yeah, that's how far I got as well. I really hoped that bootkube would mature faster and/or get absorbed into kubeadm.

What's holding this feature back from becoming a more standard practice to run k8s? It sounds like a logical next step to me to self-host the control plane after all.

[neolit123]

Yeah, that's how far I got as well. I really hoped that bootkube would mature faster and/or get absorbed into kubeadm.

i haven't heard of such plans. our current pivoting approach is pretty similar to what bootkube is doing.

What's holding this feature back from becoming a more standard practice to run k8s? It sounds like a logical next step to me to self-host the control plane after all

i do not know the answer for k8s as a whole.

for kubeadm some of the current caveats are outlined here:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#caveats

the whole idea of maintaining two different ways of cluster creation is what made us remove it from the standard init workflow.

but static Pods is also self-hosting solution, by the definition of "self-hosting" in computing.
one difference is that kubeadm calls "self-hosting" when running the CP in DaemonSets.