dockerd: failed to start daemon: Devices cgroup isn't mounted #2104

lyndalopez544 · 2019-09-24T16:52:17Z

Description

The problematic part of dockerd's output:

 
WARN[2019-09-24T16:38:43.968826308Z] Your kernel does not support cgroup memory limit 
WARN[2019-09-24T16:38:43.968902465Z] Unable to find cpu cgroup in mounts          
WARN[2019-09-24T16:38:43.968935049Z] Unable to find blkio cgroup in mounts        
WARN[2019-09-24T16:38:43.969017890Z] Unable to find cpuset cgroup in mounts       
WARN[2019-09-24T16:38:43.969031008Z] mountpoint for pids not found                
failed to start daemon: Devices cgroup isn't mounted

Output of cgroupfs-mount:

mount: /sys/fs/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/cpuset.
mount: /sys/fs/cgroup/blkio: cgroup already mounted on /sys/fs/cgroup/cpuset.
mount: /sys/fs/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/cpuset.
mount: /sys/fs/cgroup/pids: cgroup already mounted on /sys/fs/cgroup/cpuset.

Output of docker version:

Client: Docker Engine - Community
Version:           19.03.2
API version:       1.40
Go version:        go1.12.8
Git commit:        6a30dfc
Built:             Thu Aug 29 05:29:29 2019
OS/Arch:           linux/amd64
Experimental:      false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of docker info:

Client:
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

Running physically on Debian 10 with kernel 5.3.0-rc5-amd64

The text was updated successfully, but these errors were encountered:

Swiftyhu · 2019-09-26T13:57:22Z

I have the same issue.
However I found a workaround:

service docker stop
service containerd stop
cgroupfs-umount
cgroupfs-mount
service containerd start
service docker start

Sysinfo:

Linux vr 5.2.0-2-amd64 #1 SMP Debian 5.2.9-2 (2019-08-21) x86_64 GNU/Linux
cgroupfs-mount ver 1.4

containerd --version

containerd containerd.io 1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:49 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Swiftyhu · 2019-09-26T15:34:34Z

Found the "bug"...

I forgot to mention in my previous comment that I use(d) systemd version 243.
With systemd 242 works everything flawlessly... :)

From the systemd changelog:

        * systemd now defaults to the "unified" cgroup hierarchy setup during
          build-time, i.e. -Ddefault-hierarchy=unified is now the build-time
          default. Previously, -Ddefault-hierarchy=hybrid was the default. This
          change reflects the fact that cgroupsv2 support has matured
          substantially in both systemd and in the kernel, and is clearly the
          way forward. Downstream production distributions might want to
          continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for
          their builds as unfortunately the popular container managers have not
          caught up with the kernel API changes.

Sooo... Houston, we have a problem:

systemd will (or already did) jump on the cgroupsv2 bandwagon...
cgroupfs-mount tools does not work with newer systemd setups.

"Same" issue in kubernetes

kj54321 · 2019-10-30T13:41:13Z

Hi All,

 Seems issue still persist on latest Fedora 31, even with adding extra command

--exec-opt native.cgroupdriver=systemd

And per upstream workaround, which works temporarily with cgroup v1.

However, the question lays on how do we address/fix this issue permanently as it's kernel API related.

Thanks

moughamir · 2019-11-14T17:07:01Z

@kj54321
Fixed by the workaround suggested, just add the argument systemd.unified_cgroup_hierarchy=0 in grub

jnk5y · 2019-11-14T20:32:24Z

Update: installed the community edition and it works now with the added boot argument.

I added the argument but it did not solve the issue on a Fedora 31 install (upgraded from Fedora 30).

Here is my boot command line /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.3.11-300.fc31.x86_64 root=/dev/mapper/fedora_localhost--live-root ro resume=/dev/mapper/fedora_localhost--live-swap rd.lvm.lv=fedora_localhost-live/root rd.lvm.lv=fedora_localhost-live/swap rhgb quiet systemd.unified_cgroup_hierarchy=0

omercsp · 2019-11-23T12:28:06Z

@jnk5y, It does work for me on F31. Fresh install though. In the Redhat thread it looks like it worked for other people as well. Did you get the same error?

nethershaw · 2019-12-03T02:44:21Z

Obviously docker just has to support cgroupsv2.

From the cited distribution bug (which is a downstream workaround, not an upstream workaround):

cgroups v2 isn't exactly new or a surprise to anyone. Docker just dropped a ball as an upstream.

cpuguy83 · 2019-12-04T17:19:37Z

cgroups v2 support is well underway, support first has to be merged into containerd: containerd/containerd#3799

There's also some outstanding PR's on runc it seems.

whatboxapp · 2019-12-07T02:28:18Z

dnf install -y grubby

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

reboot

narfman0 · 2019-12-10T03:54:36Z

Very old fedora install here. I:

Added GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0" to /etc/default/grub
Ran grub2-mkconfig -o /boot/grub2/grub.cfg
Ran reboot

and everything was happy

duckinator · 2020-01-01T15:48:55Z

Running this on Fedora 31, after an in-place upgrade from Fedora 30, works:

sudo grubby --args='systemd.unified_cgroup_hierarchy=0' --update-kernel /boot/vmlinuz-5.3.16-300.fc31.x86_64

See a specific comment on the Fedora 31 bug report and the Fedora 31 documentation on using grubby.

arkaitzmugicaislas · 2020-01-23T21:34:35Z

I can't make it work in Fedora 31 Silverblue. I've tried everything to install and config moby-engine. Terminal copy-paste:
docker run hello-world
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/containers/create: dial unix /var/run/docker.sock: connect: permission denied.

arkaitzmugicaislas · 2020-01-23T22:27:22Z

ok... I've made it run...
The problem was on the way rpm-ostree works and how to change user groups.
grep -E '^docker:' /usr/lib/group >> /etc/group
and then
usermod -aG docker username
just that :-)

RASPWeather · 2020-02-09T16:34:17Z

Following also worked ...

dnf install -y grubby

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

reboot

jboero · 2020-02-20T11:43:30Z

This has been making my Fedora 30->31 upgrades ugly. Of course Podman works in anti-competitive efforts to push out Docker. A system-wide kernel boot option shouldn't be required for backwards compatibility with a container engine. Hmmm

jjspace · 2020-03-23T23:28:40Z

@cpuguy83 any update on this? I just ran into the issue on a new install of Fedora 31. Thankfully the workaround mentioned above worked for me. However I noticed that containerd PR was merged so I wanted to check on the progress from Docker's side.

cpuguy83 · 2020-03-23T23:48:25Z

@jjspace This will not be supported until containerd 1.4 is released. I think we have all the bits we need in dockerd for this right now, but again this depends on containerd 1.4. There are also some outstanding pieces in runc.

We are currently planning the next major release of Docker ("20.03" is the placeholder version). I do not think we will be shipping that with containerd 1.4 since that is not ready yet.... but in theory one containerd 1.4 is out once that's upgraded (along with runc) it should work.

Conan-Kudo · 2020-03-24T01:05:19Z

This has been making my Fedora 30->31 upgrades ugly. Of course Podman works in anti-competitive efforts to push out Docker. A system-wide kernel boot option shouldn't be required for backwards compatibility with a container engine. Hmmm

Seriously? The issue is that Docker has no support for cgroup v2, which is a feature all Linux distributions have had for at least five years. The fact it's now finally being adopted is pushing everyone to add support for it.

cpuguy83 · 2020-03-24T16:48:28Z

None of these comments are helpful.
This is a community effort.
Please keep politics out of it.

JiangHongTiao · 2020-04-04T07:23:52Z

Is there any update on this? Will it work for upcoming fedora 32 release out of the box?

cpuguy83 · 2020-04-04T13:40:50Z

I gave an update 12 days ago.

Will it work OOTB on Fedora 32? Not unless Fedora switches back to cgroups v1.

Support for cgroups v2 in Docker is still being worked on across the stack.

ablil · 2020-06-30T09:27:51Z

I've been struggling with this problem myself on fedora 31 for a long time.
I think you should look at the official website Fedora project

The solution suggested by @RASPWeather worked for me.

coffemate · 2020-07-23T03:23:16Z

I got another solution.
System is CentOS 6.5.
After I installed libcgroup service and started it,docker worked fine.

sudo yum install libcgroup
sudo service cgconfig start

Detail at https://www.digitalocean.com/community/tutorials/how-to-limit-resources-using-cgroups-on-centos-6

True-ps · 2021-02-22T16:15:32Z

I have the same issue.
However I found a workaround:

service docker stop
service containerd stop
cgroupfs-umount
cgroupfs-mount
service containerd start
service docker start

Sysinfo:

Linux vr 5.2.0-2-amd64 #1 SMP Debian 5.2.9-2 (2019-08-21) x86_64 GNU/Linux
cgroupfs-mount ver 1.4

containerd --version

containerd containerd.io 1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:49 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Thanks. That did the trick for me.

rajspatel1297 · 2021-07-09T17:19:08Z

I'm getting an error following error. Here is a complete output.

Setting up docker-ce (5:20.10.7~3-0~debian-buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-07-09 17:16:20 UTC; 42ms ago
     Docs: https://docs.docker.com
  Process: 15953 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 15953 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

Output of dockerd command:

INFO[2021-07-09T17:21:36.659309833Z] Starting up                                  
INFO[2021-07-09T17:21:36.665842184Z] parsed scheme: "unix"                         module=grpc
INFO[2021-07-09T17:21:36.666005302Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-07-09T17:21:36.666190669Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-07-09T17:21:36.666289415Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-07-09T17:21:36.673678480Z] parsed scheme: "unix"                         module=grpc
INFO[2021-07-09T17:21:36.673954093Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-07-09T17:21:36.674429323Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-07-09T17:21:36.674563193Z] ClientConn switching balancer to "pick_first"  module=grpc
ERRO[2021-07-09T17:21:36.692853425Z] failed to mount overlay: no such device       storage-driver=overlay2
ERRO[2021-07-09T17:21:36.693232284Z] exec: "fuse-overlayfs": executable file not found in $PATH  storage-driver=fuse-overlayfs
ERRO[2021-07-09T17:21:36.711881877Z] AUFS was not found in /proc/filesystems       storage-driver=aufs
ERRO[2021-07-09T17:21:36.725181068Z] failed to mount overlay: no such device       storage-driver=overlay
WARN[2021-07-09T17:21:36.751497714Z] Your kernel does not support cgroup memory limit 
WARN[2021-07-09T17:21:36.751650208Z] Your kernel does not support CPU CFS scheduler 
WARN[2021-07-09T17:21:36.751721705Z] Your kernel does not support CPU realtime scheduler 
WARN[2021-07-09T17:21:36.751787702Z] Unable to find blkio cgroup in mounts        
WARN[2021-07-09T17:21:36.751852199Z] Unable to find cpuset cgroup in mounts       
WARN[2021-07-09T17:21:36.751918196Z] Unable to find pids cgroup in mounts         
failed to start daemon: Devices cgroup isn't mounted

and output from docker version

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:58:04 2021
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I'm using Debian 10 (buster) on armh platform.

Thanks in advance!

curtisgray · 2021-07-10T17:26:04Z

@rajspatel1297, I used the workaround suggested by @Swiftyhu. I suspect this workaround will become obsolete at some point.

Edit:
On Debian 10 (buster) I first installed cgroupfs-mount:

sudo apt install cgroupfs-mount

thenewmantis · 2022-03-12T03:47:29Z

Don't know how helpful this will be to anyone else, but I solved the same problem by recompiling my kernel. Run contrib/check_config.sh (in the source code) to check if your kernel supports what Docker needs, recompile accordingly.

ndeloof · 2022-03-14T08:04:33Z

@thaJeztah shall we close this issue? Should be owned by https://github.com/moby/moby anyway

phisad · 2023-06-14T08:57:25Z

Came here after an update from Ubuntu 20.04 to 22.04 (still on Docker 19; cannot update easily). The workaround from Swiftyhu helped to start the deamon. Thanks!

cgroupfs-umount
cgroupfs-mount
service containerd start
service docker start

perlun · 2023-09-01T07:46:34Z

Based on #2104 (comment), the following worked for me on Ubuntu 22.04 as well (trying to run Docker-in-Docker with an old Docker 18.09.9 version).

Edited GRUB_CMDLINE_LINUX to include systemd.unified_cgroup_hierarchy=0 in /etc/default/grub
Ran sudo grub-mkconfig -o /boot/grub/grub.cfg
Ran sudo reboot

jbonhag mentioned this issue Jan 13, 2020

bash: line 4: 2412 Killed dnf -y install docker hashicorp/vagrant#11277

Closed

gavin310 mentioned this issue Mar 2, 2020

Another "failed to start daemon: Devices cgroup isn't mounted" problem on CentOS 7.7 #2369

Closed

peterwillcn mentioned this issue Aug 8, 2020

docker-ce: broken, cgroup subsystem 'devices' is missing openwrt/packages#13052

Closed

devurandom mentioned this issue Sep 16, 2020

Podman support CircleCI-Public/circleci-cli#464

Open

3 tasks

elsholz mentioned this issue Dec 15, 2020

Error during Installation: ERROR: Service 'storefront' failed to build : OCI runtime create failed saleor/saleor#6620

Closed

tstenel mentioned this issue Jan 6, 2021

Docker Fails to Start After Upgrade from 19.03.14 to 20.10.01 on Debian docker/for-linux#1177

Closed

3 tasks

elezar mentioned this issue Feb 1, 2021

cgroup issue with nvidia container runtime on Debian testing NVIDIA/nvidia-docker#1447

Closed

9 tasks

GunSik2 mentioned this issue Sep 15, 2021

remove rancher/docker GunSik2/k8s-ai#4

Open

BigBrotherJu mentioned this issue Jan 12, 2022

docker测试 ISRC-CAS/tarsier-oerv#232

Closed

ndeloof closed this as completed Mar 14, 2022

xfyan0408 mentioned this issue May 17, 2022

Errors for Devices cgroup isn't mounted when run dockerd. #3613

Closed

kamplianitis mentioned this issue Oct 4, 2022

Docker Service fails cgroupfs Debian Sid carlosedp/riscv-bringup#44

Closed

janithcooray mentioned this issue Oct 28, 2023

Package docker packages rxrbln/t2sde#88

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dockerd: failed to start daemon: Devices cgroup isn't mounted #2104

dockerd: failed to start daemon: Devices cgroup isn't mounted #2104

lyndalopez544 commented Sep 24, 2019 •

edited

Loading

Swiftyhu commented Sep 26, 2019

Swiftyhu commented Sep 26, 2019

kj54321 commented Oct 30, 2019 •

edited

Loading

moughamir commented Nov 14, 2019

jnk5y commented Nov 14, 2019 •

edited

Loading

omercsp commented Nov 23, 2019

nethershaw commented Dec 3, 2019 •

edited

Loading

cpuguy83 commented Dec 4, 2019

whatboxapp commented Dec 7, 2019 •

edited

Loading

narfman0 commented Dec 10, 2019

duckinator commented Jan 1, 2020

arkaitzmugicaislas commented Jan 23, 2020

arkaitzmugicaislas commented Jan 23, 2020

RASPWeather commented Feb 9, 2020

jboero commented Feb 20, 2020

jjspace commented Mar 23, 2020

cpuguy83 commented Mar 23, 2020

Conan-Kudo commented Mar 24, 2020

cpuguy83 commented Mar 24, 2020

JiangHongTiao commented Apr 4, 2020

cpuguy83 commented Apr 4, 2020

ablil commented Jun 30, 2020 •

edited

Loading

coffemate commented Jul 23, 2020 •

edited

Loading

True-ps commented Feb 22, 2021

rajspatel1297 commented Jul 9, 2021 •

edited

Loading

curtisgray commented Jul 10, 2021 •

edited

Loading

thenewmantis commented Mar 12, 2022

ndeloof commented Mar 14, 2022

phisad commented Jun 14, 2023 •

edited

Loading

perlun commented Sep 1, 2023 •

edited

Loading

dockerd: failed to start daemon: Devices cgroup isn't mounted #2104

dockerd: failed to start daemon: Devices cgroup isn't mounted #2104

Comments

lyndalopez544 commented Sep 24, 2019 • edited Loading

Running physically on Debian 10 with kernel 5.3.0-rc5-amd64

Swiftyhu commented Sep 26, 2019

Swiftyhu commented Sep 26, 2019

kj54321 commented Oct 30, 2019 • edited Loading

moughamir commented Nov 14, 2019

jnk5y commented Nov 14, 2019 • edited Loading

omercsp commented Nov 23, 2019

nethershaw commented Dec 3, 2019 • edited Loading

cpuguy83 commented Dec 4, 2019

whatboxapp commented Dec 7, 2019 • edited Loading

narfman0 commented Dec 10, 2019

duckinator commented Jan 1, 2020

arkaitzmugicaislas commented Jan 23, 2020

arkaitzmugicaislas commented Jan 23, 2020

RASPWeather commented Feb 9, 2020

jboero commented Feb 20, 2020

jjspace commented Mar 23, 2020

cpuguy83 commented Mar 23, 2020

Conan-Kudo commented Mar 24, 2020

cpuguy83 commented Mar 24, 2020

JiangHongTiao commented Apr 4, 2020

cpuguy83 commented Apr 4, 2020

ablil commented Jun 30, 2020 • edited Loading

coffemate commented Jul 23, 2020 • edited Loading

True-ps commented Feb 22, 2021

rajspatel1297 commented Jul 9, 2021 • edited Loading

curtisgray commented Jul 10, 2021 • edited Loading

thenewmantis commented Mar 12, 2022

ndeloof commented Mar 14, 2022

phisad commented Jun 14, 2023 • edited Loading

perlun commented Sep 1, 2023 • edited Loading

lyndalopez544 commented Sep 24, 2019 •

edited

Loading

kj54321 commented Oct 30, 2019 •

edited

Loading

jnk5y commented Nov 14, 2019 •

edited

Loading

nethershaw commented Dec 3, 2019 •

edited

Loading

whatboxapp commented Dec 7, 2019 •

edited

Loading

ablil commented Jun 30, 2020 •

edited

Loading

coffemate commented Jul 23, 2020 •

edited

Loading

rajspatel1297 commented Jul 9, 2021 •

edited

Loading

curtisgray commented Jul 10, 2021 •

edited

Loading

phisad commented Jun 14, 2023 •

edited

Loading

perlun commented Sep 1, 2023 •

edited

Loading