Rust fails to pass RPATH sanity check (and how to deal with RPATH on binary installs in general)

[casparvl]

See #18016 (comment)

Issue
libLLVM-16-rust-1.70.0-stable.so doesn't have an RPATH set. Further analysis seems to suggest it is a binary installation.

From the ./x.py build -v output:

...
download https://ci-artifacts.rust-lang.org/rustc-builds/90c541806f23a127002de5b4038be731ba1458ca/rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://ci-artifacts.rust-lang.org/rustc-builds/90c541806f23a127002de5b4038be731ba1458ca/rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz
running: "curl" "-#" "-y" "30" "-Y" "10" "--connect-timeout" "30" "--retry" "3" "-Sf" "https://ci-artifacts.rust-lang.org/rustc-builds/90c541806f23a127002de5b4038be731ba1458ca/rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz"
extracting /gpfs/nvme1/1/casparl/ebbuildpath/Rust/1.70.0/GCCcore-12.3.0/rustc-1.70.0-src/build/cache/llvm-90c541806f23a127002de5b4038be731ba1458ca-false/rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz to /gpfs/nvme1/1/casparl/ebbuildpath/Rust/1.70.0/GCCcore-12.3.0/rustc-1.70.0-src/build/x86_64-unknown-linux-gnu/ci-llvm
... 
extracting rust-dev-1.70.0-x86_64-unknown-linux-gnu/rust-dev/lib/libLLVM-16-rust-1.70.0-stable.so to /gpfs/nvme1/1/casparl/ebbuildpath/Rust/1.70.0/GCCcore-12.3.0/rustc-1.70.0-src/build/x86_64-unknown-linux-gnu/ci-llvm
...

This already suggests this particular library is just installed as a binary, and not compiled.

Comparison of the checksums of libLLVM-16-rust-1.70.0-stable.so from the downloaded tarball to the one that ends up in the installation seems to suggest that they are indeed identical:

# checksum for libLLVM-16-rust-1.70.0-stable.so from the downloaded tarball
$ cd /gpfs/nvme1/1/casparl/ebbuildpath/Rust/1.70.0/GCCcore-12.3.0/rustc-1.70.0-src/build/cache/llvm-90c541806f23a127002de5b4038be731ba1458ca-false
[casparl@tcn37 llvm-90c541806f23a127002de5b4038be731ba1458ca-false]$ ls
rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz
[casparl@tcn37 llvm-90c541806f23a127002de5b4038be731ba1458ca-false]$ tar -xf rust-dev-1.70.0-x86_64-unknown-linux-gnu.tar.xz
[casparl@tcn37 llvm-90c541806f23a127002de5b4038be731ba1458ca-false]$ cd rust-dev-1.70.0-x86_64-unknown-linux-gnu/
[casparl@tcn37 rust-dev-1.70.0-x86_64-unknown-linux-gnu]$ find -name libLLVM*
./rust-dev/lib/libLLVM-16-rust-1.70.0-stable.so
[casparl@tcn37 rust-dev-1.70.0-x86_64-unknown-linux-gnu]$ sha256sum rust-dev/lib/libLLVM-16-rust-1.70.0-stable.so
1970a0cb1b44494f6c6095341947a84bccad37654c34a724a64c98e5a4ed024e  rust-dev/lib/libLLVM-16-rust-1.70.0-stable.so

# checksum for libLLVM-16-rust-1.70.0-stable.so from the installation
$ sha256sum /scratch-nvme/1/casparl/generic/software/Rust/1.70.0-GCCcore-12.3.0/lib/libLLVM-16-rust-1.70.0-stable.so
1970a0cb1b44494f6c6095341947a84bccad37654c34a724a64c98e5a4ed024e  /scratch-nvme/1/casparl/generic/software/Rust/1.70.0-GCCcore-12.3.0/lib/libLLVM-16-rust-1.70.0-stable.so

Suggested fix / discussion
This is not the first time a binary fails the sanity check because it is simply installed. For full binary installations (i.e. using the Binary EasyBlock or anything derived from there), EasyBuild would always already skip the RPATH sanity check, simply because it could not be assumed to add an RPATH to already existing binaries (the only way EasyBuild can set an RPATH currently is through the rpath wrappers, i.e. at compile/link time).

So far, we simply accepted that binary installations did not use RPATH. According to the same rationale, we could convince EasyBuild to skip the RPATH sanity check for this Rust library. That implies that is that if sites filter LD_LIBRARY_PATH from their installed modules, this library would just use system dependencies. That is clearly undesirable, and would actually break things in the EESSI software stack, which heavily relies on RPATH.

I think we should build into EasyBuild some mechanism where we can set RPATH on existing binaries, as a postinstall-kind of step. (of course, this step should only run if build_option('rpath')). There are a number of challenges with that:

• Which patchelf should be used? Should the system provide one? Should we load a module (and if so, which version?). Technically, patchelf should then be a build dependency for any installation that requires an RPATH to be fixed. But do we really want to enforce listing patchelf in the EasyConfig for that software where it is needed (e.g. Rust), even if most people install without RPATH? Probably not. But if it's not listed in the EasyConfig, we need some other way of figuring out which version to use (maybe the global EasyBuild configuration could set it, but would you really want to point that to a versioned module? What if you build for one toolchain generation on day 1, and for another on day 2? With which toolchain should the patchelf dependency be compiled?)
• What value should be set for RPATH? We can probably peak at how we do this for the RPATH wrappers, I think it just sets anything in the LIBRARY_PATH, plus some generics like $ORIGIN, $ORIGIN/../lib and $ORIGIN/../lib64.
• Which binaries/libraries should the patchelf command set the RPATH on? We should probably have EasyBuild discover some of those automatically (e.g. all binaries and libs in <prefix/bin> and <prefix/lib> that do not have an RPATH set, need to get RPATH set. We can probably borrow logic from the RPATH sanity check here). Similarly, it should probably be easy to specify additional files on which the RPATH should be set with patchelf.

Once we figured out the 'how', my idea would be that we build an EasyBuild Framework function that takes care of this, such that it can be invoked with the right arguments from the EasyBlock, and then patchelfs the hell out of all the files it needs to patch :)

[casparvl]

Fixed by easybuilders/easybuild-easyblocks#2952