Container dont start: 'setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown'

[Dramelac]

Description

After upgrading from Debian 10 to Debian 11, i can't start any container anymore.

Steps to reproduce the issue:

1. Upgrade from Debian 10 buster to Debian 11
2. Update docker repo (buster to bullseye)
3. Upgrade packages

Describe the results you received:

sudo docker run hello-world

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown.

The container is created but not started.

sudo docker start test (Same error when starting any existing container)

Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown
Error: failed to start containers: test

Describe the results you expected:

The container should start.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:15 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:06:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 22
  Running: 0
  Paused: 0
  Stopped: 22
 Images: 136
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 4.19.0-17-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.49GiB
 Name: DRTL
 ID: 3NBB:AUYC:R2Q4:MR4C:UONQ:2ZXP:SBH5:FPJZ:Q7FP:VWZ2:U3EJ:6E2P
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: dramelac
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/16, Size: 24

WARNING: No swap limit support
WARNING: No cpuset support

Additional environment details (physical):
lsb_release -a

Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

uname -a

Linux hostname 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux

Thank you in advance for your help.

[thaJeztah]

The error itself looks to be coming from runc.

I see your kernel version is still the version from buster (4.19), or at least, I think Bullseye uses a 5.x kernel.

[Dramelac]

Indeed that was the problem!
The kernel had not been updated, fixed with:
sudo apt install linux-image-amd64

Thanks for the help!

[thaJeztah]

Good to hear!

[srd424]

(Just in case anyone else stumbles across this while googling.. I hit this error running podman containers nested inside systemd-nspawn - the issue was I needed to allow the bpf syscall in the .nspawn file.)

[theAkito]

To people coming from search engines:

I want to confirm, that updating the Kernel on my DietPi (based on Raspbian Lite) based Raspberry Pi was the only option to fix this issue.

If you are in the same situation as me, you should most likely do the following, if apt full-upgrade does not provide you with a newer version of the raspberrypi-kernel package.

1. Edit APT sources for Raspberry Pi repositories.

nano /etc/apt/sources.list.d/raspi.list

2. Update the repository you are updating from.

deb https://archive.raspberrypi.org/debian/ stretch main ui # old --> Delete this and paste the following line
deb https://archive.raspberrypi.org/debian/ bullseye main ui # new --> This should replace the old line

Of course, in your case the old version (stretch) and the new version (bullseye) may be different from this example.