runc --systemd-cgroup run: need to check that StartTransientUnit succeeds #2313
Comments
kolyshkin
changed the title
|
The issue is probably true for systemd v1 aka LegacyManager. |
This was referenced Apr 14, 2020
|
|
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container.
Moreover, since the cgroup was not created by systemd,
but the error was not detected, the container process was put
into the wrong cgroup (the same as the `crun` caller).
Finally, since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
> # cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
> 536870912
Finally, `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
> # ../crun --systemd-cgroup run -d 555
> 2020-04-16T14:47:34.000354150Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
> 2020-04-16T14:47:34.000355064Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
> # cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
> 536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
> # ../crun --systemd-cgroup run -d 555
> 2020-04-16T14:47:34.000354150Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
> 2020-04-16T14:47:34.000355064Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
> # cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
> 536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
> # ../crun --systemd-cgroup run -d 555
> 2020-04-16T14:47:34.000354150Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
> 2020-04-16T14:47:34.000355064Z: Systemd unit crun-555.scope failed: failed. See "journalctl -xe" for details.
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
[v2: also check status in destroy_systemd_cgroup_scope()]
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
# cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
# ../crun --systemd-cgroup run -d 555
2020-04-16T14:47:34.000354150Z: error creating systemd unit crun-555.scope: failed
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
[v2: also check status in destroy_systemd_cgroup_scope()]
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
# cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
# ../crun --systemd-cgroup run -d 555
2020-04-16T14:47:34.000354150Z: error creating systemd unit crun-555.scope: failed
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
While at it, abstract out the code preparing and doing the check.
[v2: also check status in destroy_systemd_cgroup_scope()]
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 16, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
# cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
# ../crun --systemd-cgroup run -d 555
2020-04-16T14:47:34.000354150Z: error creating systemd unit crun-555.scope: failed
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
While at it, abstract out the code preparing and doing the check.
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 17, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
# cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
# ../crun --systemd-cgroup run -d 555
2020-04-16T14:47:34.000354150Z: error creating systemd unit crun-555.scope: failed
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
While at it, abstract out the code preparing and doing the check.
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
to kolyshkin/crun
that referenced
this issue
Apr 17, 2020
While playing with Fedora 31 host with old/broken selinux packages,
I found out that systemd fails to create a transient unit. Here is
an except from journalctl:
> audit[1]: AVC avc: denied { setsched } for pid=1 comm="systemd" scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> systemd[1]: crun-555.scope: Failed to add PIDs to scope's control group: Permission denied
> systemd[1]: crun-555.scope: Failed with result 'resources'.
> systemd[1]: Failed to start libcrun container.
and yet crun did not show any error and proceeded to start the
container, which lead to a number of issues.
1. Since the cgroup was not created by systemd, but the error
was not detected, the container process was not put into its own
cgroup (but left in the same cgroup as the shell from which `crun`
was called).
2. Since crun gets the cgroup name from /proc/$PID/cgroup
(where $PID is container process PID), it proceeded to set the
limits for that (wrong) cgroup:
# cat /sys/fs/cgroup/system.slice/sshd.service/memory.max
536870912
3. `crun delete` apparently removes the `system.slice/sshd.service`
cgroup :(
The primary cause is the missing check that the transient unit has
been created. This is what this patch adds (similar to how it's done
in cgroup-run code).
After this patch:
# ../crun --systemd-cgroup run -d 555
2020-04-16T14:47:34.000354150Z: error creating systemd unit crun-555.scope: failed
For more background on how the issue was found, steps to repro etc
please see a similar (but much less brutal -- it just fails to start
the container) issue in runc:
- opencontainers/runc#2313
While at it, abstract out the code preparing and doing the check.
Fixes: eaccb4b
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
kolyshkin
added a commit
that referenced
this issue
Apr 28, 2020
check that StartTransientUnit/StopUnit succeeds LGTMs: @AkihiroSuda @kolyshkin Closes #2313, #2309
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR: runc should check that system was able to create a unit.
I was working on runc on a Fedora 31 Vagrant box with older selinux packages (
container-selinux-2.117.0-1.gitbfde70a.fc31.noarch). Commandrunc --systemd run xoosucceeded, but the container process was running in the same cgroup as my shell, and the designated cgroup was empty (see #2310 for details).journalctl shows:
So, systemd failed to create a unit, but runc did not show any errors (it's the same with podman+crun).
When trying to start a unit with
systemd.run, it shows an error:The text was updated successfully, but these errors were encountered: