github: switch to using 24.04 on CI runners

[simondeziel]

Current issues with the PR:

• workaround for needrestart: https://bugs.launchpad.net/ubuntu/+source/needrestart/+bug/2067800
• tests/cluster fails on 24.04 due to the kernel temporarily missing fan support (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2064508)
• tests/docker required a local workaround Apparmor rule to allow pivot_root (Ubuntu 24.04 AppArmor breaks pivot_root inside LXD containers lxd#13389)
• tests/interception fail likely due to having .ko.zst compressed (working on a potential fix)
• tests/storage-vm zfs fails due to zfs missing the rounding fix by @MggMuggins
• tests/tpm is failing due to overly long path (patch to be merged/backported, @hamistao is aware of it)

[simondeziel]

@mihalicyn the tests/docker fails due to the missing pivot_root Apparmor rule you found:

 + docker run --rm local:hello
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error jailing process inside rootfs: pivot_root .: permission denied: unknown.
Error: Command not executable
+ cleanup

Too bad I didn't try those 24.04 images earlier :)

[tomponline]

Please can we also keep testing in 22.04

[tomponline]

Could you expand comment here to explain what "deal with" means in this context and why its needed.

[simondeziel]

Please can we also keep testing in 22.04

It makes sense of course but I think we should not just multiply by 2 our CI usage, right? ATM, a (daily) test run consumes ~9h of CI.

Should we do:

• 22.04 + 24.04: latest/edge and 5.21/edge
• 22.04: 5.0/edge

[simondeziel]

After a suggestion from @mihalicyn I ran the insmod through ltrace and here's what I have after mangling the ltrace output a bit for easier diff'ing:

ltrace -s 1000 -fo /tmp/insmod.log insmod /root/nf_nat_ftp.ko:

strcmp("insmod", "kmod")                                                                                                                                               = -2
basename("insmod")                                                                                                                                                     = "insmod"
strcmp("lsmod", "insmod")                                                                                                                                              = 3
strcmp("rmmod", "insmod")                                                                                                                                              = 9
strcmp("insmod", "insmod")                                                                                                                                             = 0
getopt_long(2, 0x7ffe4a2d1578, "psfVh", 0x58a6989405c0, 0)                                                                                                             = -1
calloc(1, 144)                                                                                                                                                         = 0x58a6993b52a0
uname(0x7ffe4a2d11b0)                                                                                                                                                  = 0
__asprintf_chk(0x7ffe4a2d11a8, 2, 0x58a69893ae60, 0x58a69893ae53)                                                                                                      = 29
secure_getenv(0x58a69893b0b0, 0x7ffe4a2d0fa8, 0, 0x58a6993b5340)                                                                                                       = 0
open("/sys/module/compression", 524288, 023044730260)                                                                                                                  = 3
read(3, "zstd\n", 15)                                                                                                                                                  = 5
read(3, "", 10)                                                                                                                                                        = 0
close(3)                                                                                                                                                               = 0
strlen("modules.softdep")                                                                                                                                              = 15
malloc(32)                                                                                                                                                             = 0x58a6993b5370
__memcpy_chk(0x58a6993b5379, 0x58a69893ab92, 16, 23)                                                                                                                   = 0x58a6993b5379
malloc(24)                                                                                                                                                             = 0x58a6993b53a0
calloc(1, 64)                                                                                                                                                          = 0x58a6993b53c0
__snprintf_chk(0x7ffe4a2d0300, 4096, 2, 4096)                                                                                                                          = 45
open("/lib/modules/6.8.0-31-generic/modules.softdep", 524288, 00)                                                                                                      = -1
free(0x58a6993b5370)                                                                                                                                                   = <void>
free(0x58a6993b53a0)                                                                                                                                                   = <void>
open("/proc/cmdline", 524288, 021232311665)                                                                                                                            = 3
read(3, "BOOT_IMAGE=/vmlinuz-6.8.0-31-generic root=PARTUUID=34d8b9e2-6c7b-4efd-9f2f-68e0dc55a6d2 ro console=tty1 console=ttyS0 panic=-1\n", 4095)                      = 127
read(3, "", 3968)                                                                                                                                                      = 0
close(3)                                                                                                                                                               = 0
calloc(1, 4120)                                                                                                                                                        = 0x58a6993b5410
strdup("/root/nf_nat_ftp.ko")                                                                                                                                          = 0x58a6993b53a0
free(0)                                                                                                                                                                = <void>
stat(0x58a6993b53a0, 0x7ffe4a2d02f0, 20, 20)                                                                                                                           = 0
basename("/root/nf_nat_ftp.ko")                                                                                                                                        = "nf_nat_ftp.ko"
strlen("nf_nat_ftp")                                                                                                                                                   = 10
strlen("nf_nat_ftp")                                                                                                                                                   = 10
malloc(115)                                                                                                                                                            = 0x58a6993b6430
__memset_chk(0x58a6993b6430, 0, 104, 115)                                                                                                                              = 0x58a6993b6430
__memcpy_chk(0x58a6993b6498, 0x7ffe4a2d0380, 11, 11)                                                                                                                   = 0x58a6993b6498
strlen("nf_nat_ftp")                                                                                                                                                   = 10
realloc(0, 128)                                                                                                                                                        = 0x58a6993b64b0
calloc(1, 56)                                                                                                                                                          = 0x58a6993b6540
open("/root/nf_nat_ftp.ko", 524288, 070)                                                                                                                               = 3
read(3, "\177ELF\002\001", 6)                                                                                                                                          = 6
lseek(3, 0, 0)                                                                                                                                                         = 0
memcmp(0x7ffe4a2d12b0, 0x58a69893bc6a, 4, 0x75940971afdb)                                                                                                              = 87
memcmp(0x7ffe4a2d12b0, 0x58a69893bc64, 6, 40)                                                                                                                          = 0xffffff82
syscall(313, 3, 0x58a69893ae52, 0)                                                                                                                                     = 0
strlen("nf_nat_ftp")                                                                                                                                                   = 10
strcmp("nf_nat_ftp", "nf_nat_ftp")                                                                                                                                     = 0
memmove(0x58a6993b64b0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16)                                                                                                        = 0x58a6993b64b0
close(3)   

ltrace -s 1000 -fo /tmp/insmod-zst.log insmod /root/nf_nat_ftp.ko.zst:

strcmp("insmod", "kmod")                                                                                                                                               = -2
basename("insmod")                                                                                                                                                     = "insmod"
strcmp("lsmod", "insmod")                                                                                                                                              = 3
strcmp("rmmod", "insmod")                                                                                                                                              = 9
strcmp("insmod", "insmod")                                                                                                                                             = 0
getopt_long(2, 0x7ffde76f1db8, "psfVh", 0x56de59c2e5c0, 0)                                                                                                             = -1
calloc(1, 144)                                                                                                                                                         = 0x56de5a5c02a0
uname(0x7ffde76f19f0)                                                                                                                                                  = 0
__asprintf_chk(0x7ffde76f19e8, 2, 0x56de59c28e60, 0x56de59c28e53)                                                                                                      = 29
secure_getenv(0x56de59c290b0, 0x7ffde76f17e8, 0, 0x56de5a5c0340)                                                                                                       = 0
open("/sys/module/compression", 524288, 013160510260)                                                                                                                  = 3
read(3, "zstd\n", 15)                                                                                                                                                  = 5
read(3, "", 10)                                                                                                                                                        = 0
close(3)                                                                                                                                                               = 0
strlen("modules.softdep")                                                                                                                                              = 15
malloc(32)                                                                                                                                                             = 0x56de5a5c0370
__memcpy_chk(0x56de5a5c0379, 0x56de59c28b92, 16, 23)                                                                                                                   = 0x56de5a5c0379
malloc(24)                                                                                                                                                             = 0x56de5a5c03a0
calloc(1, 64)                                                                                                                                                          = 0x56de5a5c03c0
__snprintf_chk(0x7ffde76f0b40, 4096, 2, 4096)                                                                                                                          = 45
open("/lib/modules/6.8.0-31-generic/modules.softdep", 524288, 00)                                                                                                      = -1
free(0x56de5a5c0370)                                                                                                                                                   = <void>
free(0x56de5a5c03a0)                                                                                                                                                   = <void>
open("/proc/cmdline", 524288, 015571322700)                                                                                                                            = 3
read(3, "BOOT_IMAGE=/vmlinuz-6.8.0-31-generic root=PARTUUID=34d8b9e2-6c7b-4efd-9f2f-68e0dc55a6d2 ro console=tty1 console=ttyS0 panic=-1\n", 4095)                      = 127
read(3, "", 3968)                                                                                                                                                      = 0
close(3)                                                                                                                                                               = 0
calloc(1, 4120)                                                                                                                                                        = 0x56de5a5c0410
strdup("/root/nf_nat_ftp.ko.zst")                                                                                                                                      = 0x56de5a5c03a0
free(0)                                                                                                                                                                = <void>
stat(0x56de5a5c03a0, 0x7ffde76f0b30, 24, 24)                                                                                                                           = 0
basename("/root/nf_nat_ftp.ko.zst")                                                                                                                                    = "nf_nat_ftp.ko.zst"
strlen("nf_nat_ftp")                                                                                                                                                   = 10
strlen("nf_nat_ftp")                                                                                                                                                   = 10
malloc(115)                                                                                                                                                            = 0x56de5a5c1430
__memset_chk(0x56de5a5c1430, 0, 104, 115)                                                                                                                              = 0x56de5a5c1430
__memcpy_chk(0x56de5a5c1498, 0x7ffde76f0bc0, 11, 11)                                                                                                                   = 0x56de5a5c1498
strlen("nf_nat_ftp")                                                                                                                                                   = 10
realloc(0, 128)                                                                                                                                                        = 0x56de5a5c14b0
calloc(1, 56)                                                                                                                                                          = 0x56de5a5c1540
open("/root/nf_nat_ftp.ko.zst", 524288, 070)                                                                                                                           = 3
read(3, "(\265/\375di", 6)                                                                                                                                             = 6
lseek(3, 0, 0)                                                                                                                                                         = 0
memcmp(0x7ffde76f1af0, 0x56de59c29c6a, 4, 0x794faa51afdb)                                                                                                              = 0
syscall(313, 3, 0x56de59c28e52, 4)                                                                                                                                     = -1
__errno_location()                                                                                                                                                     = 0x794faae5a008
strerror(1)                                                                                                                                                            = "Operation not permitted"
__vasprintf_chk(0x7ffde76f1ae8, 2, 0x56de59c25860, 0x7ffde76f1af0)                                                                                                     = 73
__fprintf_chk(0x794faa6044e0, 2, 0x56de59c27f42, 0x7ffde76f26ee)                                                                                                       = 88
free(0x56de5a5c1580)                                                                                                                                                   = <void>
strlen("nf_nat_ftp")                                                                                                                                                   = 10
strcmp("nf_nat_ftp", "nf_nat_ftp")                                                                                                                                     = 0
memmove(0x56de5a5c14b0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16)                                                                                                        = 0x56de5a5c14b0
close(3)

Inside the container, it seems all the tooling is already built to support zstd compression:

root@c1:~# ll /usr/sbin/{insmod,modprobe} /usr/bin/kmod
-rwxr-xr-x 1 root root 174328 Apr 18 10:06 /usr/bin/kmod*
lrwxrwxrwx 1 root root     11 Apr 18 10:06 /usr/sbin/insmod -> ../bin/kmod*
lrwxrwxrwx 1 root root     11 Apr 18 10:06 /usr/sbin/modprobe -> ../bin/kmod*
root@c1:~# ldd /usr/bin/kmod
	linux-vdso.so.1 (0x00007ffd9e768000)
	libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007d9710467000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007d9710435000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007d970fe00000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007d970fa00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007d9710552000)

Anyway, for now my workaround of unzstd the .ko ahead of insmod'ing it works but let's see if we (/you) can figure a clean way to avoid this special handling.

[simondeziel]

@tomponline I tweaked the matrix to have 22.04 and 24.04 tested where I felt it was most important. Let me know if you wanted more coverage than this. ATM it clocks at ~11.25 hours of CI.

There are a few workarounds needed but I'll keep an eye on them and drop them ASAP.

[tomponline]

Please can we also keep testing in 22.04

It makes sense of course but I think we should not just multiply by 2 our CI usage, right? ATM, a (daily) test run consumes ~9h of CI.

Should we do:

* 22.04 + 24.04: `latest/edge` and `5.21/edge`

* 22.04: `5.0/edge`

Yep go with that.

[simondeziel]

@mihalicyn had the (clever) idea of trying to fool the insmod tool into thinking the underlying kernel didn't know how to decompress .ko.zst modules. The idea was that maybe insmod which is linked against libzstd.so would then do the decompression and then feed the .ko module to the syscall.

The way to fool the tool was to have /sys/module/compression contains nothing (instead of zstd\n). Unfortunately, that didn't work:

+ echo 'Trying with module ... nf_nat_ftp'
Trying with module ... nf_nat_ftp
+ modprobe -v nf_nat_ftp
libkmod: ERROR ../libkmod/libkmod.c:261 get_kernel_compression: unknown kernel compression insmod /lib/modules/6.8.0-31-generic/kernel/net/netfilter/nf_nat_ftp.ko.zst 
+ rmmod nf_nat_ftp
libkmod: ERROR ../libkmod/libkmod.c:261 get_kernel_compression: unknown kernel compression + lsmod
+ grep nf_nat_ftp
libkmod: ERROR ../libkmod/libkmod.c:261 get_kernel_compression: unknown kernel compression + true
++ modinfo -n nf_nat_ftp
libkmod: ERROR ../libkmod/libkmod.c:261 get_kernel_compression: unknown kernel compression + MODULE_PATH=/lib/modules/6.8.0-31-generic/kernel/net/netfilter/nf_nat_ftp.ko.zst
++ basename /lib/modules/6.8.0-31-generic/kernel/net/netfilter/nf_nat_ftp.ko.zst
+ MODULE_FILE_NAME=nf_nat_ftp.ko.zst
+ lxc file push /lib/modules/6.8.0-31-generic/kernel/net/netfilter/nf_nat_ftp.ko.zst c1/root/
+ lxc exec c1 -- insmod /root/nf_nat_ftp.ko.zst                                                                                                   
insmod: ERROR: could not insert module /root/nf_nat_ftp.ko.zst: Operation not permitted
+ lxc config set c1 linux.kernel_modules.load=ondemand
+ lxc restart c1 -f
+ lxc exec c1 -- touch /root/empty
+ lxc exec c1 -- mount -o ro,bind /root/empty /sys/module/compression
+ lxc exec c1 -- insmod /root/nf_nat_ftp.ko.zst
libkmod: ERROR ../libkmod/libkmod.c:261 get_kernel_compression: unknown kernel compression insmod: ERROR: could not insert module /root/nf_nat_ftp.ko.zst: Function not implemented
+ lxc config set c1 linux.kernel_modules=nf_nat_ftp
+ lxc restart c1 -f
+ lxc exec c1 -- insmod /root/nf_nat_ftp.ko.zst
insmod: ERROR: could not insert module /root/nf_nat_ftp.ko.zst: Operation not permitted
+ cleanup
+ set +e
+ echo ''

+ '[' 1 = 1 ']'
+ echo 'Test failed'

For the time being, I'll keep the existing workaround of doing ahead of time manual decompression (unzstd) of the module before doing the insmod.

Thanks Aleks, I learned a ton while looking into this with you and trying to workaround the issue!