zfs breaks nvidia-container-toolkit

[hansom82]

Both extensions uses shared library libc.so. If both present on target host then nvidia-device-plugin crashes with error:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 127, stdout: , stderr: Auto-detected mode as 'legacy'
/usr/local/bin/nvidia-container-cli: error while loading shared libraries: /usr/local/glibc/lib/libc.so: invalid ELF header: unknown

Environment

• Talos version: v1.7.1
• Kubernetes version: 1.28.7
• Platform: metal (Proxmox and Bare Metal)

/fyi @kvaps

[jahanson]

This has caused an issue for me as well

The libc.so seems to be an ASCII file when most are symbolic links to their partner .so.# files.

amd-ucode                  20240410
nonfree-kmod-nvidia        535.129.03-v1.7.1
nvidia-container-toolkit   535.129.03-v1.14.6
zfs                        2.2.3-v1.7.1

[frezbo]

This is interesting though, zfs should be linked against the musl libc, so i guess the zfs extensions ship some extra libc, which it shouldn't

[frezbo]

This is due to both nvidia and zfs needing libtirpc and the zfs libtirpc is linked against musl while the nvidia one is linked against glibc. These two extensions are mutually incompatible with each other for now

[rdegez]

As heavy zfs users and soon-to-be nvidia extension users as well we would be glad providing assistance to help solving this one.

@frezbo what do you think would be the best way to fix this then ?

Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?

Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?

From the libtirpc source package, the autoconf snippet handling libc detection is located in in libtirpc-1.3.4/config.guess :

<...>
case $UNAME_SYSTEM in
Linux|GNU|GNU/*)
        LIBC=unknown

        set_cc_for_build
        cat <<-EOF > "$dummy.c"
        #include <features.h>
        #if defined(__UCLIBC__)
        LIBC=uclibc
        #elif defined(__dietlibc__)
        LIBC=dietlibc
        #elif defined(__GLIBC__)
        LIBC=gnu
        #else
        #include <stdarg.h>
        /* First heuristic to detect musl libc.  */
        #ifdef __DEFINED_va_list
        LIBC=musl
        #endif
        #endif
        EOF
        cc_set_libc=`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g'`
        eval "$cc_set_libc"

        # Second heuristic to detect musl libc.
        if [ "$LIBC" = unknown ] &&
           command -v ldd >/dev/null &&
           ldd --version 2>&1 | grep -q ^musl; then
                LIBC=musl
        fi

        # If the system lacks a compiler, then just pick glibc.
        # We could probably try harder.
        if [ "$LIBC" = unknown ]; then
                LIBC=gnu
        fi
        ;;
esac
<...>

[frezbo]

@rdegez

Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?

We do not want to do this, makes it a crazy hard to follow build steps

Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?

Nvidia has strict requirement for glibc and musl will not work at all

What we were discussing internally is to setup some LD_ options to let zfs use the right libtirpc

[rdegez]

@rdegez

Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?

We do not want to do this, makes it a crazy hard to follow build steps

Looks like a lot of hassle Indeed...

Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?

Nvidia has strict requirement for glibc and musl will not work at all

Yeah... i'm not so surprised about that.

What we were discussing internally is to setup some LD_ options to let zfs use the right libtirpc

Like an alternate $LD_LIBRARY_PATH ?

What about using https://github.com/NixOS/patchelf ?
I wasn't aware of this (quite recent) trick 5 until minutes ago but I find it quite elegant and useful.

See https://stackoverflow.com/a/44710599 or https://www.baeldung.com/linux/multiple-glibc

[frezbo]

I want to avoid patchelf, at some point the whole nvidia thing was using a lot of patchelf to fix stuff, it just makes it harder for others and for us internally to keep things up to date, I'll be looking at this issue next week and see what would work great for us and easy to maintain, don't want to make it more complicated

[rdegez]

Fair enough :-)
Feel free to bug me here or on slack if I can be of any use to you!

[jahanson]

@frezbo Will this fix be possible on 1.7 or will we have to wait for 1.8?

[frezbo]

@frezbo Will this fix be possible on 1.7 or will we have to wait for 1.8?

Sorry, probably only for 1.8, found other issues and fixing them, don't want to backport such major fixes