fix: glibc search paths for nvidia #421
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
frezbo
force-pushed
the
fix/nvidia-add-new-search-path
branch
from
cafe267 to
24c76e0
Compare
frezbo
commented
Jun 24, 2024
| @@ -1,6 +1 @@ | |||
| # libc default configuration | |||
There was a problem hiding this comment.
This is to make sure, glibc doesn't try to load non-glibc libs
frezbo
commented
Jun 24, 2024
| @@ -51,7 +51,7 @@ steps: | |||
| cd libnvidia-container | |||
|
|
|||
| # LDLIBS=-L/usr/local/glibc/lib is set so that libnvidia-container-cli libs which are hardcoded as -llibname and not using pkg-config | |||
| CPPFLAGS="-I/usr/local/glibc/include/tirpc" LDLIBS="-L/usr/local/glibc/lib -ltirpc -lelf -lseccomp" make | |||
| CPPFLAGS="-I/usr/local/glibc/include/tirpc" LDLIBS="-L/usr/local/glibc/lib -ltirpc -lelf -lseccomp" LDFLAGS='-Wl,--rpath=\$$ORIGIN/../glibc/\$$LIB' make | |||
There was a problem hiding this comment.
This is the actual fix needed
frezbo
commented
Jun 24, 2024
| cd NVIDIA-Linux-* | ||
|
|
||
| ./nvidia-installer --silent \ | ||
| --opengl-prefix=/rootfs/usr/local \ | ||
| --utility-prefix=/rootfs/usr/local \ | ||
| --utility-libdir=glibc/lib \ |
There was a problem hiding this comment.
this will only copy some libraries, not all hence the manual move below at https://github.com/siderolabs/extensions/pull/421/files#diff-d91dc320bde5625cb12535516b84cc3cd36e7ae24b320a40852b8f22a38299beR61
There was a problem hiding this comment.
for some reason nvidia-container-cli can't find libs stored under /usr/local/lib so we explicitly keep them under the custom location at /usr/local/glibc/lib
Set `glibc/lib` as first `rpath` for `nvidia-container-cli`. Also install nvidia libraries to `/usr/local/glibc/lib` so any musl libraries lives separately. `nvidia-container-cli` explicitly sets an `RPATH` as `$ORIGIN/../$LIB` here: https://gitlab.com/nvidia/container-toolkit/libnvidia-container/-/blob/v1.14.6/Makefile?ref_type=tags#L183, this means `/usr/local/lib` would be searched first, since `zfs` and nvidia ship their own `libtirpc`, `nvidia-container-cli` first tries to use the `libtirpc` shippeed with `zfs` at `/usr/local/lib` instead of the one at `/usr/local/glibc/lib`. Fix this by setting an additional `RPATH` as `$ORIGIN/../glibc/$LIB`, so that libraries in `/usr/local/glibc/lib` have higher preference. ```bash ❯ scanelf -r _out/rootfs/rootfs/usr/local/bin/nvidia-container-cli TYPE RPATH FILE ET_DYN $ORIGIN/../glibc/$LIB:$ORIGIN/../$LIB _out/rootfs/rootfs/usr/local/bin/nvidia-container-cli ``` Properly fixes: siderolabs#380 Fixes from siderolabs#401 and siderolabs#410 were not complete. Manually tested by spinning up a NVIDIA worker in AWS. Signed-off-by: Noel Georgi <git@frezbo.dev>
frezbo
force-pushed
the
fix/nvidia-add-new-search-path
branch
from
24c76e0 to
5334e89
Compare
|
/m |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Set
glibc/libas firstrpathfornvidia-container-cli. Also install nvidia libraries to/usr/local/glibc/libso any musl libraries lives separately.nvidia-container-cliexplicitly sets anRPATHas$ORIGIN/../$LIBhere: https://gitlab.com/nvidia/container-toolkit/libnvidia-container/-/blob/v1.14.6/Makefile?ref_type=tags#L183, this means/usr/local/libwould be searched first, sincezfsand nvidia ship their ownlibtirpc,nvidia-container-clifirst tries to use thelibtirpcshippeed withzfsat/usr/local/libinstead of the one at/usr/local/glibc/lib. Fix this by setting an additionalRPATHas$ORIGIN/../glibc/$LIB, so that libraries in/usr/local/glibc/libhave higher preference.Properly fixes: #380
Fixes from #401 and #410 were not complete.
Manually tested by spinning up a NVIDIA worker in AWS.