nixos/k3s: enable enableUnifiedCgroupHierarchy

[euank]

Motivation for this change

NixOS switched the default cgroup driver forward to cgroupsv2 a bit ago. At the time, k3s wasn't ready for it. The bug preventing using unified cgroups with k3s + docker has since been fixed, so we should be able to roll forward!

The k3s tests are happy with this change, which gives me decent confidence it works, and moving forward to cgroupsv2 is of course desirable.

Note, this PR is carrying forward #134365; Thanks for the original pr @ngerstle, appreciated!

Things done

• Built on platform(s)

  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin

• Tested, as applicable:

  • NixOS test(s)
    Yes, both tests pass:
    • nix-build -A nixosTests.k3s-single-node-docker
    • nix-build -A nixosTests.k3s-single-node

• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage

• Tested basic functionality of all binary files (usually in ./result/bin/)

• 22.05 Release Notes (or backporting 21.11 Release notes)

  • (Module updates) Added a release notes entry if the change is significant
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

• Fits CONTRIBUTING.md.

[Congee]

LGTM. Will this also be backported to 21.11?

[euank]

For good measure, I moved forward one of my worker nodes to this branch too, and it's running along happily with it.

---

Will this also be backported to 21.11?

We could backport it if there's breaking issues in 21.11, but I'm not convinced it's worth it currently.

I understand from your comment on the other issue that k3s check-config complains about cgroups v1, but I don't actually have any evidence that it breaks anything (and my nodes with cgroupsv1 have been fine), and I don't think check-config on its own is something worth backporting for.

It's also possible to get this behavior on 21.11 already I think with:

# The k3s module should have used mkDefault, but we can still mkForce over it I think
systemd.enableUnifiedCgroupHierarchy = mkForce true;
services.k3s.extraFlags = "--kubelet-arg=cgroup-driver=systemd"

What do you think? Are there issues other than the check-config output complaining? Do you think that's enough to merit a backport?

[Congee]

Honestly, I don't know, as I am still unable to get k3s running.

check-config actually complains about cgroupsv2 and running k3s server gave me an error failed to find cpu cgroup (v2). However, after rebooting, cgroupv2 related error is gone, but, I see another cgroup related error that may not be related to this PR.

"Failed to get the kubelet's cgroup. Kubelet system container metrics may be missing." err="mountpoint for cpu not found"

I've been using the xanmod 5.15.2 kernel with NixOS 21.11. It seems ok to not backport, though.

[superherointj]

Honestly, I don't know, as I am still unable to get k3s running.

check-config actually complains about cgroupsv2 and running k3s server gave me an error failed to find cpu cgroup (v2). However, after rebooting, cgroupv2 related error is gone, but, I see another cgroup related error that may not be related to this PR.

"Failed to get the kubelet's cgroup. Kubelet system container metrics may be missing." err="mountpoint for cpu not found"

I've been using the xanmod 5.15.2 kernel with NixOS 21.11. It seems ok to not backport, though.

This PR targets master. Test with master please.

[Congee]

Honestly, I don't know, as I am still unable to get k3s running.
check-config actually complains about cgroupsv2 and running k3s server gave me an error failed to find cpu cgroup (v2). However, after rebooting, cgroupv2 related error is gone, but, I see another cgroup related error that may not be related to this PR.

"Failed to get the kubelet's cgroup. Kubelet system container metrics may be missing." err="mountpoint for cpu not found"

I've been using the xanmod 5.15.2 kernel with NixOS 21.11. It seems ok to not backport, though.

This PR targets master. Test with master please.

Sorry didn't make it clear. That was about backporting to 21.11. So, I have to test with 21.11 without this PR.

@euank Just tested with linuxPackages_latest instead of the xanmod kernel in 21.11. k3s works. So no backport is needed at least in my case.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/451