PSA: runc 1.0.0-rc93 / containerd 1.4.4 bug

[champtar]

Hi All,

the runc included in containerd.io 1.4.4 seems to have a "fun" bug leading to "PLEG is not healthy"
https://support.rancher.com/hc/en-us/articles/360058516672-containerd-io-1-4-4-bug-advisory#how-do-i-know-if-i-am-impacted--0-0
containerd/containerd#5261
opencontainers/runc#2871

When using containerd we might be ok as we now use io.containerd.runc.v2 but docker uses shim v1 and is impacted according to my reading

[champtar]

1.4.6 is out https://github.com/containerd/containerd/releases/tag/v1.4.6

[stefan-kolev]

Do you plan to update containerd to v1.4.6 in Release v2.16?

[olevitt]

Here is a "fix" we came up with.
Basically it updates containerd and runc and then reboots.
Not sure how dirty this is and if I understood everything correctly, use at your own risk !
From what I understood, this fix is only needed if you stay on kubespray v2.16 (as later versions will bundle a more recent containerd / runc) and obviously if you are using docker as engine.
Feel free to comment if this looks weird to you or you feel I missed the point of this issue.

#!/bin/sh
# Fix for issue https://github.com/kubernetes-sigs/kubespray/issues/7574
wget https://github.com/containerd/containerd/releases/download/v1.4.6/containerd-1.4.6-linux-amd64.tar.gz
tar xvf containerd-1.4.6-linux-amd64.tar.gz
mkdir backup-containerd
cp /usr/bin/containerd backup-containerd/
cp /usr/bin/containerd-shim backup-containerd/
cp /usr/bin/containerd-shim-runc-v1 backup-containerd/
cp /usr/bin/containerd-shim-runc-v2 backup-containerd/
cp /usr/bin/ctr backup-containerd/
rm /usr/bin/containerd
rm /usr/bin/containerd-shim
rm /usr/bin/containerd-shim-runc-v1
rm /usr/bin/containerd-shim-runc-v2
rm /usr/bin/ctr
cp bin/containerd /usr/bin
cp bin/containerd-shim /usr/bin
cp bin/containerd-shim-runc-v1 /usr/bin
cp bin/containerd-shim-runc-v2 /usr/bin
cp bin/ctr /usr/bin
wget https://github.com/opencontainers/runc/releases/download/v1.0.0-rc95/runc.amd64
cp /usr/bin/runc backup-containerd/
rm /usr/bin/runc
cp runc.amd64 /usr/bin/runc
chmod +x /usr/bin/runc
reboot

[tongpu]

We just upgraded to containerd v1.4.9 on Ubuntu by adding the required key to containerd_versioned_pkg in roles/container-engine/containerd-common/vars/ubuntu.yml. Could this be a change that would warrant the release of v2.16.1?

[rptaylor]

Yes a new release would be good, Kubespray 2.16.0 should not be used in production with this issue.

containerd/containerd#5274