Doc tests fail when a library is built with LTO

[cuviper]

Problem

When a library is built with LTO enabled, doctests fail with linker errors, "file format not recognized".

This worked with Rust/Cargo 1.45.2, but fails with 1.46 through nightly. This might be a rustdoc issue instead, but I'm starting here because of the Cargo 1.46 changes in #8349.

In the example below, I turned on LTO in profile.dev, which I know is unusual. In practice, we saw this in a Fedora crate build that had LTO in profile.release. We always build --release for the distro, and then we test --release too so we don't build everything twice.

Steps

1. Cargo.toml:

[package]
name = "foo"
version = "0.1.0"
authors = ["Josh Stone <cuviper@gmail.com>"]
edition = "2018"

[profile.dev]
lto = true

2. src/lib.rs:

/// ```
/// foo::foo()
/// ```
pub fn foo() {}

3. Test docs:

$ cargo +stable test --doc --verbose
   Compiling foo v0.1.0 (/tmp/foo)
     Running `rustc --crate-name foo --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -Clinker-plugin-lto -C debuginfo=2 -C metadata=aabfeee930865724 -C extra-filename=-aabfeee930865724 --out-dir /tmp/foo/target/debug/deps -C incremental=/tmp/foo/target/debug/incremental -L dependency=/tmp/foo/target/debug/deps`
    Finished test [unoptimized + debuginfo] target(s) in 0.08s
   Doc-tests foo
     Running `rustdoc --edition=2018 --crate-type lib --test /tmp/foo/src/lib.rs --crate-name foo -L dependency=/tmp/foo/target/debug/deps -L dependency=/tmp/foo/target/debug/deps --extern foo=/tmp/foo/target/debug/deps/libfoo-aabfeee930865724.rlib`

running 1 test
test src/lib.rs - foo (line 1) ... FAILED

failures:

---- src/lib.rs - foo (line 1) stdout ----
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/rustdoctestmicFHJ/rust_out.rust_out.7rcbfp3g-cgu.0.rcgu.o" "-o" "/tmp/rustdoctestmicFHJ/rust_out" "/tmp/rustdoctestmicFHJ/rust_out.33dyzt1ekirinwy8.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/tmp/foo/target/debug/deps" "-L" "/tmp/foo/target/debug/deps" "-L" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/tmp/foo/target/debug/deps/libfoo-aabfeee930865724.rlib" "-Wl,--start-group" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-cf0f33af3a901778.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-daf8c2d692e6eca4.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-24e8f97647425e48.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-85ed7d2b484c05a9.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-89de2c581262ec09.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-3b0db98e62ed7d75.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-c60847f9a163de82.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-0bb9b63424f4fc5d.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-3f74d829e37fa40e.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0e9d83ff06f1a7ad.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-2c8c904efaf7c40b.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-cbfb51de52131460.rlib" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-97497c26fddb7882.rlib" "-Wl,--end-group" "/home/jistone/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-f1a9d8c443e20b5e.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"
  = note: /usr/bin/ld: /tmp/foo/target/debug/deps/libfoo-aabfeee930865724.rlib: error adding symbols: file format not recognized
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

Couldn't compile the test.

failures:
    src/lib.rs - foo (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

Possible Solution(s)

Notes

Output of cargo version:

$ cargo +stable -Vv
cargo 1.46.0 (149022b1d 2020-07-17)
release: 1.46.0
commit-hash: 149022b1d8f382e69c1616f6a46b69ebf59e2dea
commit-date: 2020-07-17

[ehuss]

Thanks for the detailed report! This is a duplicate of #8654, and has been fixed via #8657, which should land in nightly sometime next week. I have also proposed backporting to beta in #8658.

[cuviper]

Ah, thank you! I should have searched closed issues first. I'll see if I can grab that fix for our distro toolchain.