Kubelet: invalid configuration: cgroup-root ["kubelet"] doesn't exist

[aojea]

What happened:

Running a node-image built with kind 92f54895, in github actions, using kind 0.9.0 makes kubelet fail to start with the following error:

Dec 07 19:16:44 ovn-control-plane kubelet[994]: F1207 19:16:44.170173 994 server.go:269] failed to run Kubelet: invalid configuration: cgroup-root ["kubelet"] doesn't exist
Dec 07 19:16:44 ovn-control-plane kubelet[994]: goroutine 1 [running]:

How to reproduce it (as minimally and precisely as possible):

I've tried to reproduced it locally without success:

./kind.v0.9.0 create cluster --image aojea/kindnode:kind92f54895_k8se1c617a88ec

The failing job with the errors and logs is here
https://github.com/aojea/ovn-kubernetes/runs/1512853588?check_suite_focus=true

Bear in mind that github actions have its own docker and we had issues with KIND running there before

Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., 0.4.2+azure)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 19.03.13+azure
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1031-azure
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.791GiB
 Name: fv-az183-477
 ID: ZDHP:ZO2E:H6PK:BNIM:UC36:GA3L:TTM6:4HUU:GWDU:GGRT:AMDI:I2E7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[aojea]

We have a repro and a job to test we don´t regress #1965 (comment)

[BenTheElder]

This really doesnt make sense. git diff HEAD..v0.9.0 -- ./pkg/cluster/internal/providers/docker/ is only network changes.

The image changed, yes. But changing kind binary versions shouldn't affect it, we basically haven't changed how we run nodes (except network setup) 🤔

[BenTheElder]

I've been comparing the container inspect.jsons from the various CI results. Nothing shows up.

[BenTheElder]

I can't repro locally:

[bentheelder@bentheelder:~/go/src/sigs.k8s.io/kind·2020-12-11T22:23:32-0800·@0ff469a7]
$ kind --version
kind version 0.9.0
[bentheelder@bentheelder:~/go/src/sigs.k8s.io/kind·2020-12-11T22:25:03-0800·@0ff469a7]
$ docker exec kind-control-plane ps aux | grep cgroup-root
root         688  3.7  0.1 1874900 89188 ?       Ssl  06:22   0:05 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --fail-swap-on=false --node-ip=172.17.0.3 --provider-id=kind://docker/kind/kind-control-plane --fail-swap-on=false --cgroup-root=/kubelet

[aojea]

There are similar issues in kubernetes , it has to be related to cgroups

kubernetes/kubernetes#75832
kubernetes/kubernetes#49281
kubernetes/kubernetes#92175

[BenTheElder]

those issues reference this flag but I'm not sure they're really related.

[BenTheElder]

so I wasn't testing with 1.20 though (kubernetes HEAD is a bit ahead of that), probably not related but something to confirm.

EDIT: still works fine locally

[aojea]

so I wasn't testing with 1.20 though (kubernetes HEAD is a bit ahead of that), probably not related but something to confirm.

EDIT: still works fine locally

I´ve added more verbosity to the CI job and we have more info

Dec 15 11:07:14 kind-control-plane kubelet[978]: I1215 11:07:14.725431 978 server.go:470] Sending events to api server.
Dec 15 11:07:14 kind-control-plane kubelet[978]: I1215 11:07:14.725896 978 cgroup_manager_linux.go:294] The Cgroup [kubelet] has some missing paths: [/sys/fs/cgroup/hugetlb/kubelet /sys/fs/cgroup/cpu,cpuacct/kubelet /sys/fs/cgroup/memory/kubelet /sys/fs/cgroup/pids/kubelet /sys/fs/cgroup/cpuset/kubelet /sys/fs/cgroup/cpu,cpuacct/kubelet /sys/fs/cgroup/systemd/kubelet]
Dec 15 11:07:14 kind-control-plane kubelet[978]: F1215 11:07:14.725994 978 server.go:269] failed to run Kubelet: invalid configuration: cgroup-root ["kubelet"] doesn't exist
Dec 15 11:07:14 kind-control-plane kubelet[978]: goroutine 1 [running]:

[aojea]

ok, we are pregressing, it turns out is not an issue with the kind version that creates the cluster, it fails with all 😄

https://github.com/kubernetes-sigs/kind/runs/1557595731?check_suite_focus=true

[BenTheElder]

fix inbound :-)

[BenTheElder]

@aojea can you confirm this is resolved in your the downstream environment?

[AkihiroSuda]

Which k/k commit is related to this?

[BenTheElder]

@AkihiroSuda this was a regression in a different iteration of the entrypoint / cgroups changes in this repo IIRC, not Kubernetes version or the kind binary.

[BenTheElder]

aside: github adds a lot of visual noise when an issue is referenced in a commit, it will add an event to the issue every time any form of the commit is pushed/merged in any fork, including people merging in forks etc.

in kubernetes/kubernetes commits doing this are actually banned by automation.
https://github.com/kubernetes/test-infra/blob/55dab213bff9ed663013072c14f17c19dc9c625f/prow/plugins/invalidcommitmsg/invalidcommitmsg_test.go#L55

[AkihiroSuda]

Sorry for noises 🙇

[BenTheElder]

No worries, more of an FYI than anything, I think kind is small enough to not have major issues with it anyhow, but I've seen it get pretty out of hand on Kubernetes issues 🙃

That comment / code covers some other fun ones, the most ridiculous one on github's part IMHO is @user in commits actually causing an email to the user under the same circumstances (~push/merge in any fork) for users with typical notification settings 🔥

[luhongzhen317]

Has the problem been solved?I have a similar problem

[YanzhaoLi]

Hit the same issue. But I checked my entrypoint which has already included the fix https://github.com/kubernetes-sigs/kind/pull/1995/files.

I have two kind clusters, one works well but another one hit this issue.