k3s doesn't work with unified cgroups and the docker runtime

[euank]

Describe the bug

nixpkgs has moved to unified cgroups (woo 🎉) as of recently (#73800).

Unfortunately, k3s fails to start correctly with unified cgroups without k3s-io/k3s#2844, which isn't included in a release yet.

Until that lands in a release and nixpkgs is updated, it can be worked around by setting systemd.enableUnifiedCgroupHierarchy = false;

I intend to work on fixing this; I'll drop any notes along the way in this issue.

[euank]

I've updated the title to be more specific, since my actual errors are coming from using k3s with services.k3s.docker = true;, and the error is specifically about kubelet and docker's cgroup driver mismatching.

I see that the unified cgroup PR did disable unified cgroups in the k3s module, so that probably should work before adding in the docker runtime.
I need to do a little more investigation here.

[pinpox]

Please let me know, if this is related:
I'm trying to follow the wiki article on kubernetes on nixos: https://nixos.wiki/wiki/Kubernetes
With the configs shown there I run into an error when starting kubelet.service

failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

The full error is very long, I've uploaded it here: https://gist.github.com/pinpox/4058c540a42d47f456c2e87d5ac3f856

[lbodor]

Thanks for looking into this.

[pinpox]

Just tried the workaround mentioned above: Setting
systemd.enableUnifiedCgroupHierarchy = false; also fixes the problem for the kubernetes service

[imw]

We have observed that unified cgroups also breaks Kubernetes-in-Docker for node images prior to 1.19. The observed behavior is that the kubelet fails to start, with the message:failed to run Kubelet: mountpoint for not found

Also able to work around with systemd.enableUnifiedCgroupHierarchy = false;

[euank]

For k3s, the v1.20.4+k3s1 release includes unified cgroups support. It fixes my use of k3s, though I also had to set a flag in the nixos module (--kubelet-arg="cgroup-driver=systemd" to make docker+kubelet match).

I've opened up a PR to bump the package (#114215), and I'll make a separate PR to fix the nixos module to pass that arg.
I think those two changes together should fix this issue.

It sounds like there's other work to be done for other ways of running kubernetes on nixos too, but I think we should open separate issues for those to avoid conflating too many issues at once.

[Mic92]

This would be the fix: #111590
I was not invested enough at the time to also write release notes, but if you want you can pick up the change.

[Mic92]

see #118801

[06kellyjac]

This PR should also help: k3s-io/k3s#3242

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-setup-kubernetes-k3d-on-nixos/13574/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/declarative-rootless-k3s/49839/1