cpu.cfs_quota_us: permission denied + mkdir issue on debian jessie #57

lukas2511 · 2015-06-27T17:35:24Z

I can't get this to run on my debian jessie machine(s) because of (seemingly) multiple issues.

I created rootfs with docker export $(docker create busybox) > busybox.tar; mkdir rootfs; tar -C rootfs -xf busybox.tar

After creating container.json with runc spec | sed 's/daemon/root/' > container.json i get the following error running runc:

WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: open /sys/fs/cgroup/cpu/user.slice/test/cpu.cfs_quota_us: permission denied

If i run it again the error changes:

WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: mkdir : no such file or directory

runc --debug output looks exactly the same. No more details.

Running strace i can see that the code actually does try to use mkdir("") after reading an empty string from /proc/self/mountinfo:

427 open("/proc/self/mountinfo", O_RDONLY|O_CLOEXEC) = 25
428 read(25, "14 19 0:14 / /sys rw,nosuid,node"..., 4096) = 2919
429 read(25, "", 4096)                      = 0
430 stat("", 0xc2080b06c0)                  = -1 ENOENT (No such file or directory)
431 mkdir("", 0755)                         = -1 ENOENT (No such file or directory)
432 lstat("", 0xc2080b0750)                 = -1 ENOENT (No such file or directory)

If i remove the cpu (and memory) definitions from container.json i get that mkdir error directly.

The text was updated successfully, but these errors were encountered:

avagin · 2015-06-30T11:52:28Z

I think here are two different problems. I'm going to investigate the second problem with mkdir. Could you apply this patch and show output?

diff --git a/libcontainer/cgroups/fs/apply_raw.go b/libcontainer/cgroups/fs/apply_raw.go
index 3d099d8..de730d2 100644
--- a/libcontainer/cgroups/fs/apply_raw.go
+++ b/libcontainer/cgroups/fs/apply_raw.go
@@ -11,6 +11,7 @@ import (
        "strconv"
        "sync"

+       "github.com/Sirupsen/logrus"
        "github.com/opencontainers/runc/libcontainer/cgroups"
        "github.com/opencontainers/runc/libcontainer/configs"
 )
@@ -102,6 +103,7 @@ func (m *Manager) Apply(pid int) error {
                }
        }()
        for name, sys := range subsystems {
+               logrus.Debugf("Apply configuration for %s", name)
                if err := sys.Apply(d); err != nil {
                        return err
                }

jkahrs · 2015-06-30T13:28:14Z

@avagin I'm getting the same error. Output after Patch:

DEBU[0000] Apply configuration for net_prio             
DEBU[0000] Apply configuration for perf_event           
DEBU[0000] Apply configuration for devices              
WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: mkdir : no such file or directory

edit: Missed the update to the container spec. Now the above Debug messages are gone. However, the following remains:

WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: mkdir : no such file or directory

lukas2511 · 2015-06-30T13:52:49Z

If i run it with this patch I get a few different outputs every other run:

DEBU[0000] Apply configuration for net_cls              
DEBU[0000] Apply configuration for net_prio             
DEBU[0000] Apply configuration for perf_event           
DEBU[0000] Apply configuration for cpu                  
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:49:04.210696523 +0200 CEST
Code: System error

Message: open /sys/fs/cgroup/cpu/user.slice/testbla/cpu.cfs_quota_us: permission denied

DEBU[0000] Apply configuration for cpuset               
DEBU[0000] Apply configuration for blkio                
DEBU[0000] Apply configuration for memory               
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:49:47.228252894 +0200 CEST
Code: System error

Message: mkdir : no such file or directory

DEBU[0000] Apply configuration for hugetlb              
DEBU[0000] Apply configuration for net_prio             
DEBU[0000] Apply configuration for freezer              
DEBU[0000] Apply configuration for memory               
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:49:58.24831853 +0200 CEST
Code: System error

Message: mkdir : no such file or directory

DEBU[0000] Apply configuration for net_prio             
DEBU[0000] Apply configuration for perf_event           
DEBU[0000] Apply configuration for devices              
DEBU[0000] Apply configuration for cpu                  
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:50:11.455998621 +0200 CEST
Code: System error

Message: open /sys/fs/cgroup/cpu/user.slice/testbla/cpu.cfs_quota_us: permission denied

DEBU[0000] Apply configuration for cpuacct              
DEBU[0000] Apply configuration for net_cls              
DEBU[0000] Apply configuration for perf_event           
DEBU[0000] Apply configuration for devices              
DEBU[0000] Apply configuration for cpu                  
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:50:29.479062715 +0200 CEST
Code: System error

Message: open /sys/fs/cgroup/cpu/user.slice/testbla/cpu.cfs_quota_us: permission denied

DEBU[0000] Apply configuration for memory               
WARN[0000] signal: killed                               
Timestamp: 2015-06-30 15:52:02.252456041 +0200 CEST
Code: System error

Message: mkdir : no such file or directory

At least those with the mkdir error seem to have in common that they all are applying the memory stuff before it crashes.

lukas2511 · 2015-06-30T14:20:29Z

Just a note: On my systems I didn't set the "cgroup_enable=memory swapaccount=1" options that seem to be required for memory limiting, maybe that has something to do with this problem.

avagin · 2015-06-30T16:22:01Z

Could you attache output for

cat /proc/self/cgroup
cat /proc/self/mountinfo

lukas2511 · 2015-06-30T16:29:02Z

cgroup

8:perf_event:/
7:blkio:/user.slice
6:net_cls,net_prio:/
5:freezer:/
4:devices:/user.slice
3:cpu,cpuacct:/user.slice
2:cpuset:/
1:name=systemd:/user.slice/user-0.slice/session-c218.scope

mountinfo

14 19 0:14 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
15 19 0:3 / /proc rw,relatime shared:12 - proc proc rw,gid=27,hidepid=2
16 19 0:5 / /dev rw,relatime shared:2 - devtmpfs udev rw,size=10240k,nr_inodes=4103425,mode=755
17 16 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
18 19 0:15 / /run rw,nosuid,relatime shared:5 - tmpfs tmpfs rw,size=6569412k,mode=755
19 0 0:16 / / rw,relatime shared:1 - btrfs /dev/mapper/box--vg-root rw,space_cache
20 14 0:18 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
21 16 0:19 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
22 18 0:20 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
23 14 0:21 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
24 23 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
25 14 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:11 - pstore pstore rw
26 23 0:24 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,cpuset
27 23 0:25 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,cpu,cpuacct
28 23 0:26 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,devices
29 23 0:27 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,freezer
30 23 0:28 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,net_cls,net_prio
31 23 0:29 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,blkio
32 23 0:30 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,perf_event
33 15 0:31 / /proc/sys/fs/binfmt_misc rw,relatime shared:20 - autofs systemd-1 rw,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
34 16 0:32 / /dev/hugepages rw,relatime shared:21 - hugetlbfs hugetlbfs rw
35 16 0:13 / /dev/mqueue rw,relatime shared:22 - mqueue mqueue rw
36 14 0:6 / /sys/kernel/debug rw,relatime shared:23 - debugfs debugfs rw
37 19 0:34 / /var/tmp rw,relatime shared:24 - tmpfs none rw
38 19 0:35 / /tmp rw,relatime shared:25 - tmpfs none rw
39 19 8:1 / /boot rw,relatime shared:26 - ext2 /dev/sda1 rw,stripe=64
40 18 0:36 / /run/rpc_pipefs rw,relatime shared:27 - rpc_pipefs rpc_pipefs rw
238 18 0:73 / /run/user/1000 rw,nosuid,nodev,relatime shared:28 - tmpfs tmpfs rw,size=3284708k,mode=700,uid=1000,gid=1000
303 33 0:79 / /proc/sys/fs/binfmt_misc rw,relatime shared:73 - binfmt_misc binfmt_misc rw
309 18 0:80 / /run/user/0 rw,nosuid,nodev,relatime shared:76 - tmpfs tmpfs rw,size=3284708k,mode=700
315 19 0:16 /var/lib/docker/btrfs /var/lib/docker/btrfs rw,relatime - btrfs /dev/mapper/box--vg-root rw,space_cache

avagin · 2015-06-30T18:08:35Z

You need to mount the memroy controller:
mount -t cgroup -o memory memory /sys/fs/cgroup/memory/

avagin · 2015-06-30T18:12:08Z

As for cfs_quota_us, you probably use a kernel where this parameter isn't supported

ls -l /sys/fs/cgroup/cpu/cpu.cfs_quota_us

lukas2511 · 2015-06-30T19:22:20Z

Mounting using that command doesn't work.

mount: special device memory does not exist

Even if those two things aren't supported, runc shouldn't fail with an mkdir failure. Ideally it shouldn't fail at all.

jkahrs · 2015-07-01T07:53:37Z

@lukas2511
You can enable memory support via grub.config.
The option for this is:

GRUB_CMDLINE_LINUX="cgroup_enable=memory"

Remember to update-grub2 afterwards

lukas2511 · 2015-07-01T14:38:44Z

@jkahrs
But I don't want to enable it.

avagin · 2015-07-02T12:38:56Z

@lukas2511 We have optional controllers (e.g. hugetlb), so I think it's possible to make the memory controller optional too, if memory limits are not set.

LK4D4 · 2015-07-02T17:09:50Z

I agree with @avagin . Memory shouldn't be mandatory.

Fixes: opencontainers#57 Normally all cgroup subsystems are optional except device cgroup, but memory cgroup optional was broken by: docker-archive/libcontainer#637 This patch fixes this. Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>

hqhq · 2015-07-13T11:24:30Z

Normally all cgroup subsystems are optional except device cgroup, but memory cgroup optional was broken by: docker-archive/libcontainer#637
I've sent a PR to fix this, #118 .

LK4D4 · 2015-07-13T17:23:22Z

Closed in #118

hqhq mentioned this issue Jul 13, 2015

Fix error when memory cgroup not mounted #118

Merged

LK4D4 closed this as completed Jul 13, 2015

haircommander mentioned this issue Sep 30, 2020

rasbian support in obs cri-o/cri-o#4183

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cpu.cfs_quota_us: permission denied + mkdir issue on debian jessie #57

cpu.cfs_quota_us: permission denied + mkdir issue on debian jessie #57

lukas2511 commented Jun 27, 2015

avagin commented Jun 30, 2015

jkahrs commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

avagin commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

avagin commented Jun 30, 2015

avagin commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

jkahrs commented Jul 1, 2015

lukas2511 commented Jul 1, 2015

avagin commented Jul 2, 2015

LK4D4 commented Jul 2, 2015

hqhq commented Jul 13, 2015

LK4D4 commented Jul 13, 2015

cpu.cfs_quota_us: permission denied + mkdir issue on debian jessie #57

cpu.cfs_quota_us: permission denied + mkdir issue on debian jessie #57

Comments

lukas2511 commented Jun 27, 2015

avagin commented Jun 30, 2015

jkahrs commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

avagin commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

avagin commented Jun 30, 2015

avagin commented Jun 30, 2015

lukas2511 commented Jun 30, 2015

jkahrs commented Jul 1, 2015

lukas2511 commented Jul 1, 2015

avagin commented Jul 2, 2015

LK4D4 commented Jul 2, 2015

hqhq commented Jul 13, 2015

LK4D4 commented Jul 13, 2015