Using dylib as dependency without -Cprefer-dynamic doesn't work

[bjorn3]

I tried this:

//- a.rs
pub fn foo() {}

//- b.rs
extern crate a;
fn main() {
    a::foo();
}

$ rustc a.rs --crate-type dylib
$ rustc b.rs --crate-type bin --extern a

I expected to see this happen: It compiles and b dynamically links to liba.so which itself statically links to libstd.

Instead, this happened:

error: cannot satisfy dependencies so `std` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `core` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `compiler_builtins` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_std_workspace_core` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `alloc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `libc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `unwind` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `cfg_if` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `hashbrown` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_std_workspace_alloc` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `rustc_demangle` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `std_detect` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `addr2line` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `gimli` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `object` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `memchr` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `panic_unwind` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: aborting due to 17 previous errors

Making it work currently requires passing -Cprefer-dynamic when compiling liba.so.

Meta

rustc --version --verbose:

rustc 1.56.0 (09c42c458 2021-10-18)
binary: rustc
commit-hash: 09c42c45858d5f3aedfa670698275303a3d19afa
commit-date: 2021-10-18
host: x86_64-unknown-linux-gnu
release: 1.56.0
LLVM version: 13.0.0

This seems to have been the reason for a new test in #89836 failing.

[crazyboycjr]

This issue has been one year and I can confirm it with a recent compiler (rustc 1.67.0-nightly (09508489e 2022-11-04)).

$ rustc b.rs --crate-type bin --extern a=liba.so
error: cannot satisfy dependencies so `std` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: aborting due to previous error

What's the current state of the issue (about the expected behavior or any plan to fix it)?

Also, I will be grateful if anyone could point me to the source code for the 'fundamental' reason of not being able to have the same dependency show up more than once. I've been stuck by various linkage issues about dynamic libraries in rust for weeks. I've seen people discuss this problem around (#34909 #3573) but could never figure out the underlying reason. @bjorn3 @alexcrichton

[bjorn3]

The same crate not being allowed to show up in more than one linked artifact (dylib or executable) is deliberate. Duplication will lead to either symbol conflicts or symbols being duplicated. Neither of which is acceptable as it will break the guarantee that statics are never duplicated and with it break any crate using global state, including the standard library itself.

This issue however is about rustc deciding to dynamically link libstd despite the fact that it is already included in the dylib. This is something that can and should be fixed.

[crazyboycjr]

Now I get it. This sounds quite reasonable and convincing. I also saw the following description about library duplication in this RFC just now.

In the common case duplication of a library in the same process does not tend to have adverse side effects, but some of the more flavorful features tend to interact adversely with duplication such as:

• Globals with significant addresses (statics). These globals would all be duplicated and have different addresses depending on what library you're talking to.
• TLS/TLD. Any "thread local" or "task local" notion will be duplicated across each library in the process.

Thanks so much for your explanation @bjorn3!

[DemiMarie]

What if libstd is a dylib?

[bjorn3]

The problem is exactly that libstd is available as both an rlib and dylib. Without -Cprefer-dynamic rustc will pick the rlib and link it into the user dylib and then when attempting to link against this user dylib, it will try to pick the dylib version of libstd instead of the already linked in rlib version. If libstd was only available as either rlib or dylib there shouldn't be any issue.

[lvella]

This is a rustc bug or a cargo bug? I.e., should this intelligence of deciding whether to use -Cprefer-dynamic be part of rustc or cargo? There is this related cago issue.

What if the dylib has other common dependencies with the binary, won't it cause the same issue?

[bjorn3]

Cargo already passes -Cprefer-dynamic for compiling everything when any crate depended on by your code is only available as dylib. It doesn't do so when compiling a dylib as final artifact though.

This issue is about a rustc bug.