Init KEP to migrate in-tree dockershim to out-of-tree #866
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| --- | ||
| title: Separating a CRI for docker from Kubelet | ||
| authors: | ||
| - "@resouer" | ||
| - "@dims" | ||
| - "@zhangxiaoyu-zidif" | ||
| owning-sig: sig-node | ||
| reviewers: | ||
| - "@yujuhong" | ||
| - "@dchen1107" | ||
| - "@derekwaynecarr" | ||
| - "@PatrickLang" | ||
| approvers: | ||
| - "@DawnChen" | ||
| - "@yujuhong" | ||
| creation-date: 2019-02-26 | ||
| last-updated: 2019-02-26 | ||
| status: provisional | ||
| --- | ||
|
|
||
| # Separating a CRI for Docker from Kubelet | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Terms](#terms) | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| * [Pros](#pros) | ||
| * [Cons](#cons) | ||
| * [Goals](#goals) | ||
| * [Non-Goals](#non-goals) | ||
| - [Proposal](#proposal) | ||
| * [Dockershim deprecation plan](#dockershim-deprecation-plan) | ||
| * [Dockershim deprecation criteria](#dockershim-deprecation-criteria) | ||
| * [Test Plan](#test-plan) | ||
| * [Graduation Criteria](#graduation-criteria) | ||
| - [Implementation History](#implementation-history) | ||
|
|
||
| ## Terms | ||
|
|
||
| - **CRI:** Container Runtime Interface – a plugin interface which enables kubelet to use a wide variety of container runtimes, without the need to recompile. | ||
|
|
||
|
|
||
| ## Summary | ||
|
|
||
| CRI for docker (i.e. dockershim) is currently a built-in container runtime in kubelet code base. This proposal aims at a concrete deprecation and migration plan for separating dockershim from kubelet to out-of-tree without breaking current production users and WIP engineering efforts. | ||
|
|
||
| ## Motivation | ||
|
|
||
| In Kubernetes, CRI is the used as the "default" container runtime, while currently the CRI of docker (a.k.a. dockershim) is part of kubelet code and coupled with kubelet's lifecycle. | ||
|
|
||
| This is not ideal as kubelet then has dependency on specific container runtime which leads to maintenance burden for not only developers in sig-node, but also cluster administrators when critical issues (e.g. runc CVE) happen to container runtimes. The pros of moving dockershim to out-of-tree is straightforward: | ||
|
|
||
| ### Pros | ||
| - Docker is not special and just a CRI just like every other CRI in our ecosystem. | ||
| - Currently, dockershim "enjoys" some backdoors for various reasons. Deprecating these "features" should eliminate maintenance burden of kubelet. | ||
| - A cri-dockerd can be maintained independently. | ||
| - Over time we can remove vendored docker dependencies in kubelet. | ||
|
|
||
| Having said that, cons of deprecation built-in dockershim requires lots of attentions: | ||
|
|
||
| ### Cons | ||
| - Deployment pain with a new binary in addition to kubelet. | ||
| - An additional component may aggravate the complexity currently. It may be relieved with docker version evolutions. | ||
| - The number of affected users maybe large. | ||
|
There was a problem hiding this comment. I'd be more specific about how this affects the users. |
||
| - Users must change existing use experience when using Kubernetes and docker. | ||
| - Users have to change their existing workflows to adapt to this new changes. | ||
| - And other unrecorded stuff. | ||
| - Updating all the eco-system tools to support the new cri-dockerd. | ||
| - Many people use the built in dockershim for in-cluster image build. While that may not be something we recommend for a variety of reasons, it will be a breaking change for these users. | ||
| - CRI is still in alpha，should probably get a 1.0 out there splitting out dockershim completely from kubelet. | ||
|
There was a problem hiding this comment. I strongly support progressing CRI to v1, for the sake of all the runtime implementations. I'm less clear on the benefit of extracting dockershim to a standalone CRI. Did you consider deprecating dockershim in favor of existing CRI implementations instead? There was a problem hiding this comment. I will try to confirm if sig-node or Docker Inc has a plan to support a CRI-docker for long time (which is what I heard during the sig meeting). If not, deprecating dockershim directly will be the goal. There was a problem hiding this comment. We talked about this at SIG Node, including the folks from Docker Inc. Docker folks plan to maintain cri-dockershim. |
||
| - Existing CNI and CSI plugins may also be affected. | ||
|
There was a problem hiding this comment. Could you be more specific about how these plugins are going to be affected? There was a problem hiding this comment. For one, every container runtime generates CNI calls slightly differently. It might be nice to standardize (even if not officially) on one of the k8s cni wrappers, be it oci-cni or the existing dockershim networking plugin. Also, is this the time to finally decomission kubenet? |
||
| - Current dockershim has independent module interacting with CNI plugins. After migrating dockershim out of Kubelet, it may affect some processes between dockershim and CNI plugins. | ||
| - cri-dockerd will vendor kubernetes/kubernetes, that may be tough. | ||
|
There was a problem hiding this comment. For cri-dockerd, I think you should also include the memory overhead for separating dockershim out from kubelet. There was a problem hiding this comment. yes, independent software need extra resource allocation. |
||
| - cri-dockerd as an independent software running on node should be allocated enough resource to guarantee its availability. | ||
|
|
||
|
There was a problem hiding this comment. cri-dockerd will still be vendoring kubernetes/kubernetes for a while during the transition. vendoring kubernetes/kubernetes is not easy |
||
| > You can check [the discussion in sig-node mailing list](https://groups.google.com/forum/#!msg/kubernetes-sig-node/0qVzfugYhro/l6Au216XAgAJ) for more details. | ||
|
|
||
| Based on all the discussion, we agree that we should not rush to immediate decision. At the same time, it's the right time to start designing and documenting dockershim deprecation criteria and plan, which will be the main content of rest of this KEP. | ||
|
|
||
| ### Goals | ||
|
|
||
| - A concrete dockershim deprecation criteria. | ||
| - A brief plan to deprecate dockershim spanning multiple releases. | ||
|
|
||
|
There was a problem hiding this comment. Can we add another goal of writing up a criteria for adopting one of the existing CRI runtimes (for example containerd) that can eventually replace cri-dockerd? There was a problem hiding this comment. considering that there are too many docker users, it be better not to cease cri-dockerd here. There was a problem hiding this comment. SIG Node doesn't want to make a call which CRI implementation the vendor / the user should use. Instead, we design the API to define how the runtime engaging with Kubelet and nodes, a test suite to verify / qualify the functionalities, a portable toolset (cri-tool) for debuggability and introspect. |
||
| ### Non-Goals | ||
|
|
||
| - Deprecation of dockershim immediately without consideration for users and WIP efforts depending on it. | ||
| - Refactoring or re-design of dockershim itself due to deprecation. | ||
|
|
||
| ## Proposal | ||
|
|
||
| ### Dockershim deprecation criteria | ||
|
|
||
| - CRI itself is beta. | ||
| - kubelet has no dependency on dockershim/docker in its whole lifecycle. | ||
| - All node related features are CRI generic and have no "back door" dependency on dockershim/docker. | ||
|
There was a problem hiding this comment. Deprecate and remove, or replace all Docker-specific features. |
||
| - Deprecate and remove, or replace all Docker-specific features. | ||
| - Reasonable benchmark result of performance degradation after moving dockershim to out-of-tree. | ||
|
There was a problem hiding this comment. What percentage of degradation is acceptable (10%) ? There was a problem hiding this comment. 10% might not be acceptable by the wider audience. |
||
| - A out-of-tree CRI for docker is implemented and well maintained, and become to beta. | ||
| - E2E test framework has been updated with fully support of out-of-tree CRI container runtime. | ||
|
|
||
| ### Dockershim deprecation plan | ||
|
|
||
| Step 1: Stabilize in-tree dockershim and decouple dockershim from kubelet (but still in-tree). | ||
|
|
||
| Target releases: 1.15, 1,16, 1.17 | ||
|
|
||
|
|
||
| Actions: | ||
|
|
||
| - Mark in-tree dockershim as "maintenance mode": | ||
| - CRI generic changes/features can continue on dockershim. | ||
| - WIP efforts on dockershim can continue and go to complete. | ||
| - dockershim/docker specific changes/features should be rejected. | ||
|
There was a problem hiding this comment. we will continue updating the docker/docker dependency. right? There was a problem hiding this comment. Perhaps continue updating the docker/docker dependency is a must. There was a problem hiding this comment. Yes, we still need to update Docker when needed. |
||
| - Deprecate the legacy features of dockershim in kubelet by providing a specific timeline. Currently, kubelet still has: | ||
| - vendored dockershim | ||
| - flags that are used to configure dockershim. | ||
| - support to get container logs when docker uses journald as the driver. | ||
| - logic of moving docker processes to a given cgroup | ||
| - TBD anything else? | ||
| - Package in-tree dockershim is separated from kubelet and provide a "option" to enable/disable it. And the original in-tree dockershim will be remained there currently and depreciated gradually. | ||
|
|
||
| - Ensure e2e/Node e2e test framework is CRI generic and test cases are independent of container runtime. | ||
|
|
||
| Step 2: Work out a out-of-tree CRI for docker | ||
|
There was a problem hiding this comment. does this not already exist in the form of containerd? There was a problem hiding this comment. I assume they are different. During recent sig-node meeting, folks from Docker Inc are willing to maintain a CRI-docker. There was a problem hiding this comment. All recent versions of docker are built on top of containerd, so I'm unclear on what functionality dockershim has that cri-containerd does not. @Random-Liu do you know if there is any reason to use dockershim over cri-containerd with recent versions of docker? There was a problem hiding this comment. I think in the long run, we do expect people to move to CRI conformant container runtimes like containerd and cri-o to be able to leverage new Kubernetes features. However, people may still need more time to make that move, one of the biggest reason is that with dockershim, you can actually see all your Kubernetes containers in Docker, and people's tools may have been relying on that. For example, people may have built security tools, monitoring tools, logging tools around that. With containerd or cri-o, you won't see Kubernetes containers in Docker any more. It takes time for people to migrate their tooling, that's why we need to keep dockershim for a while. There was a problem hiding this comment. Another reason is that today's Kubernetes windows support is still based on dockershim. cri-containerd Windows is WIP, but not finished yet. There was a problem hiding this comment. so we cloud remain both mechanisms currently and depreciated the old way gradually. There was a problem hiding this comment. Yes, there are still features that are supported by docker/dockershim, but not with containerd. |
||
|
|
||
| Target releases: 1.18 | ||
|
|
||
| Actions: | ||
|
|
||
| - Design & implement a out-of-tree CRI for docker, it can be "copied" from dockershim as beginning. | ||
|
There was a problem hiding this comment. Why do we want to start a new repo before eliminating the in-tree dockershim completely? What benefits does this bring when we have to maintain two copies for multiple releases? There was a problem hiding this comment. I wish we could stop shipping new features (except critical bugfix) to build-in dockershim as early as possible, we could mark it as "deprecated" status and lock to specific version, in this case, new CRI functions should return "Not implemented". Otherwise, I'm afraid the deprecating process would be blocked by upcoming features in dockershim. |
||
| - Re-direct dockershim related features/changes to this out-of-tree CRI for docker. | ||
|
|
||
|
|
||
| Step 3: Completely deprecate in-tree dockershim from kubelet. | ||
|
|
||
| Target releases: TBD, we probably need to continue keeping in-tree dockershim for 3 releases as grace period. | ||
|
|
||
| Actions: | ||
|
|
||
| - Refactoring e2e/Node e2e test framework to include CRI for docker installation (or use other CRI container runtime). | ||
|
There was a problem hiding this comment. this would be extremely helpful for existing CRI impls to test/verify compatibility There was a problem hiding this comment. Cluster/node e2e doesn't care what CRI implementation you use under the hood. I think maybe I've missed the point of this statement. There was a problem hiding this comment. There're two points in my mind:
|
||
| - Ensure cluster/node e2e are 100% CRI focused. | ||
| - Ensure test-infra install CRI-docker or Containerd binary in e2e machines. Currently, they install Docker only. | ||
| - Document and announce migration guide. | ||
| - Delete in-tree dockershim code from kubelet after certain "grace period". | ||
|
|
||
|
|
||
| ### Test Plan | ||
|
|
||
| _To be filled until targeted at a release._ | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| _To be filled until targeted at a release._ | ||
|
|
||
| ## Implementation History | ||
|
|
||
| - 2019-02-28: Initial KEP sent out for discussion & reviewing. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/@DawnChen/@dchen1107?