Fix native program loading when solana-runtime is a git or crates.io dependency

[jstarry]

Problem

The native loader runtime looks for dynamic libraries with a specific filename like this: libsolana_bpf_loader_program.dylib and has been working fine when the runtime dependency is a relative path but breaks in other scenarios. Cargo seems to add an extra hash suffix to dependency filenames when pulling in dependencies from git and crates.io

Proposed Solution

Handle hash suffixed dynamic library file names like this: libsolana_bpf_loader_program-dad324483fd13565.dylib

[mvines]

Cargo seems to add an extra hash suffix to dependency filenames when pulling in dependencies from git and crates.io

What's an example of where/when this happens in practice? This almost feels like a "you're holding it wrong" kind of problem -- eg, stop using a native program, use BPF instead.

[jstarry]

@mvines it happens when testing bpf programs with the solana runtime in a different project. For example, this line was breaking because solana-runtime is a git dependency: https://github.com/solana-labs/solana-voib-demo/pull/10/files#diff-b8c3f81279869c0ce939d0328874bca3R218

Seeing the hash in solana-voib-demo/target/deps:

But not in solana/programs/bpf/target/deps:

[mvines]

It's not a problem if it's a crates.io dependency though right?

[jstarry]

Just double checked and it is still a problem when using crates.io

[mvines]

Ah, I see. From cargo’s perspective there could be many versions of that crate in the build. We’re asking it to do something it’s not equipped for. We really want to cargo install those native program shared objects into a known location, but not sure we can get there without hacks because `cargo installs only works for bins.

[jstarry]

@mvines I'm using this hacky workaround for now here: https://github.com/solana-labs/solana-voib-demo/pull/10/files#diff-5e76fee89c5d06973f4f702bc6b21a65R27. We definitely want something better for developers who wish to test their programs using the runtime in the future though.

[mvines]

For this kind of local testing we could statically link the native programs into the runtime, there's really no need for the dynamic loading at all

[jstarry]

@mvines good point, I was running into issues with that approach but have a fix here: #5992

[mvines]

@jstarry - think we can call this issue resolved with #5992?

[jstarry]

Sounds good to me. Closing because there's a better approach unblocked by #5992

[jstarry]

Re-opening to avoid this hacky workaround: #6383

[jackcmay]

Looks like the consistent messaging from the Rust folks is explicit dynamic linking of direct dependencies is not expected, encouraged, nor supported. Instead the recommendation is to treat these dependencies like any other FFI dependency and pull them in as separate projects (using build scripts for example). We can still keep the programs as direct dependencies for cases where we statically link them (treated as libs by runtime) but I didn't find any viable workarounds to address the file name hash while keeping the dynamically linked programs as direct dependencies.

If we treat the dynamically linked programs as external dependencies I see two options:

• Pull the programs in as git submodules. This would mean for example that TDS add a submodule for any programs that exist as stand-alone git repo and/or adding the Solana repo as a submodule for programs that live there
• Use build.rs to clone, build, and install programs into a known location (similar result as the first option)

Not super excited about either of these but they would address this issue. If we went down this path we might also want to split the programs living in Solana out to their own (or a single) repo.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This stale issue has been automatically closed. Thank you for your contributions.