Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

oci runtime error: no space left on device #642

Closed
jovizhangwei opened this issue Sep 28, 2017 · 4 comments · Fixed by #643
Closed

oci runtime error: no space left on device #642

jovizhangwei opened this issue Sep 28, 2017 · 4 comments · Fixed by #643
Assignees
@sameo

Comments

@jovizhangwei
Copy link

jovizhangwei commented Sep 28, 2017 •
edited
Loading

I found below error in CC 3.0.0.

$sudo docker run -it --rm --runtime=runc centos pwd
/

$sudo docker run -it --rm centos bash
docker: Error response from daemon: oci runtime error: write /sys/fs/cgroup/cpu/docker/ff666f1149c9fd19ed254564b412086d1dbf2ed24d29262fac8946c4decd5a4f/tasks: no space left on device.

$sudo cc-runtime cc-env
[Meta]
  Version = "1.0.3"

[Runtime]
  [Runtime.Version]
    Semver = "3.0.0"
    Commit = "7fc11d5"
    OCI = "1.0.0-rc5"
  [Runtime.Config]
    GlobalLogPath = "/var/lib/clear-containers/runtime/runtime.log"
    [Runtime.Config.Location]
      Path = "/usr/share/defaults/clear-containers/configuration.toml"
      Resolved = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  [Hypervisor.Location]
    Path = "/usr/bin/qemu-system-x86_64"
    Resolved = "/usr/bin/qemu-system-x86_64"

[Image]
  Path = "/usr/share/clear-containers/clear-containers.img"
  Resolved = "/usr/share/clear-containers/clear-17270-containers.img"

[Kernel]
  Parameters = ""
  [Kernel.Location]
    Path = "/usr/share/clear-containers/vmlinuz.container"
    Resolved = "/usr/share/clear-containers/vmlinuz-4.9.47-77.2.container"

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.0+git.c66f21e"
  URL = "unix:///var/run/clear-containers/proxy.sock"

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.0-alpha.4 (commit: 739e5e2)"
  [Shim.Location]
    Path = "/usr/libexec/clear-containers/cc-shim"
    Resolved = "/usr/libexec/clear-containers/cc-shim"

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"
  [Agent.PauseBin]
    Path = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"
    Resolved = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"

[Host]
  Kernel = "3.10.0-693.2.2.el7.x86_64"
  CCCapable = false
  [Host.Distro]
    Name = "Alibaba Group Enterprise Linux Server"
    Version = "7.2"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz"
@jovizhangwei jovizhangwei mentioned this issue Sep 28, 2017
Open
@jovizhangwei
Copy link
Author

Ping @sameo, @jodh-intel

@Ace-Tang
Copy link
Contributor

the reason is here opencontainers/runc#133

I fix this by simply copy cpuset.cpus and cpuset.mems from it's parent cgroup path in the code

@jovizhangwei
Copy link
Author

Thanks @Ace-Tang

So I guess it's kernel's problem, but is it possible cc-runtime can fix it just like runc did?

@Ace-Tang
Copy link
Contributor

#643
@jovizhangwei

@Ace-Tang Ace-Tang mentioned this issue Sep 28, 2017
Merged
sameo pushed a commit that referenced this issue Sep 28, 2017
@Ace-Tang
cgroups: set value for cpuset if it's empty
8b569e1 
fix issue when echo "$id" > /sys/fs/cgroup/cpu/docker/$container-id/tasks
will get error message: no space left on device, this is because
cpuset.cpus and cpuset.mems in the container's cpu cgroup
directory is empty, the following test can explain this:

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs
bash: echo: write error: No space left on device

set value in cpuset.cpus and cpuset.mems can fix the problem

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cgroup.procs
1

this patch fix above issue by copy data from cpu cgroup
root directory to container's cpuset.cpus and cpuset.mems

Fixes #642.

Signed-off-by: Ace-Tang <aceapril@126.com>
@sameo sameo mentioned this issue Sep 28, 2017
Closed
@sameo sameo self-assigned this Sep 28, 2017
sameo pushed a commit that referenced this issue Sep 29, 2017
@Ace-Tang
cgroups: set value for cpuset if it's empty
7ae0574 
fix issue when echo "$id" > /sys/fs/cgroup/cpu/docker/$container-id/tasks
will get error message: no space left on device, this is because
cpuset.cpus and cpuset.mems in the container's cpu cgroup
directory is empty, the following test can explain this:

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs
bash: echo: write error: No space left on device

set value in cpuset.cpus and cpuset.mems can fix the problem

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cgroup.procs
1

this patch fix above issue by copy data from cpu cgroup
root directory to container's cpuset.cpus and cpuset.mems

Fixes #642.

Signed-off-by: Ace-Tang <aceapril@126.com>
@sameo sameo mentioned this issue Sep 29, 2017
Closed
sameo pushed a commit that referenced this issue Sep 29, 2017
@Ace-Tang
cgroups: set value for cpuset if it's empty
f6717a8 
fix issue when echo "$id" > /sys/fs/cgroup/cpu/docker/$container-id/tasks
will get error message: no space left on device, this is because
cpuset.cpus and cpuset.mems in the container's cpu cgroup
directory is empty, the following test can explain this:

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs
bash: echo: write error: No space left on device

set value in cpuset.cpus and cpuset.mems can fix the problem

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.mems

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo "0-1" > footest/cpuset.cpus

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ echo 1 > footest/cgroup.procs

[root@r10e19288.sqa.zmf /sys/fs/cgroup/cpu]
$ cat footest/cgroup.procs
1

this patch fix above issue by copy data from cpu cgroup
root directory to container's cpuset.cpus and cpuset.mems

Fixes #642.

Signed-off-by: Ace-Tang <aceapril@126.com>
@sameo sameo removed the in progress label Sep 30, 2017
mcastelino pushed a commit to mcastelino/runtime that referenced this issue Dec 6, 2018
@jodh-intel
versions: Bump golang from 1.8.3 to 1.9.7
d814bc5 
golang version 1.8.3 is old and the runtime cannot even be built with
it now it seems.

Since it is no longer considered a stable version [1], move to the
oldest official stable version (version 1.9.7).

Fixes clearcontainers#642.

[1] - https://golang.org/dl/

Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@sameo sameo

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@sameo @Ace-Tang @jovizhangwei