Nightly CI failed: collect2: fatal error: cannot find 'ld'

[RalfJung]

I have no idea what's going on: the test crate doesn't build any more in the cronjob, with a linker error that makes no sense.

   Compiling test-cargo-careful v0.1.0 (/home/runner/work/cargo-careful/cargo-careful/test)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcm63Rj1/symbols.o" "/home/runner/work/cargo-careful/cargo-careful/test/target/debug/deps/test_cargo_careful-0e47937edafe05ad.test_cargo_careful.3e635eeb7087b08-cgu.0.rcgu.o" "/home/runner/work/cargo-careful/cargo-careful/test/target/debug/deps/test_cargo_careful-0e47937edafe05ad.612wei78qutq3859040oig1g4.rcgu.o" "-Wl,--as-needed" "-L" "/home/runner/work/cargo-careful/cargo-careful/test/target/debug/deps" "-L" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9850d55ffa95b2c1.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-c06d8630b9bd3e1a.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-717303c051853526.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-b7d4d45c43d06a82.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-5d5fb08222d1873a.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-712ef105c182535c.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-12b169224999bec3.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-e4b909410881a022.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-aafa0da98621d9f0.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-d28c65980cadfb76.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-042fde5f8a770748.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-9878825974fe8303.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-daa4f21143c253f3.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-bf95eb0232141247.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-3f47c3c5e46fa734.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b536590ed87e8bd7.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-72d5c488a18d4c75.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-7d5185bea9a378ab.rlib" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-8a033d7396b3726f.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/lib/self-contained" "-o" "/home/runner/work/cargo-careful/cargo-careful/test/target/debug/deps/test_cargo_careful-0e47937edafe05ad" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: collect2: fatal error: cannot find 'ld'
          compilation terminated.

[RalfJung]

nightly-2024-05-18 is the first bad nightly, nightly-2024-05-17 still works.
Sounds like it would probably be caused by rust-lang/rust#124129.

@lqd any idea why that would break the build here...? I'm not doing anything weird wrt linking that I am aware of. I am building custom sysroot (which still works fine) and then building a crate against that, and then somehow that can't find the linker?!?

[RalfJung]

My main confusion is why it is even trying to run ld in the first place. Or is that just really poor error reporting from cc where it doesn't say which command it actually tried to run?

The other odd thing is the paths cargo/rustc have set:

PATH="/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:...

and also "-B/home/runner/.cache/cargo-careful/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld".
There is no /home/runner/.cache/cargo-careful/lib/....

It seems almost as if it is taking the sysroot path and then walking the file system from there, but that's entirely wrong when the sysroot is overwritten with a locally built one.

[lqd]

It seems almost as if it is taking the sysroot path and then walking the file system from there, but that's entirely wrong when the sysroot is overwritten with a locally built one.

Yeah, rust-lld is supposed to be in the tools search paths, and rustc adds them to the linker invocation. You can revert back to the default linker with -Zlinker-features=-lld while we look into it, or if it's at build time with --set rust.lld = false when you use ./x (or via config.toml).

rust-lld has been present in the sysroot for a long time and is used by default by some other targets, so I wonder if the sysroot here is just incomplete.

[RalfJung]

so I wonder if the sysroot here is just incomplete.

That's possible -- I assumed a sysroot is complete by just filling it with the output of the standard library build. That worked so far 🤷

[lqd]

I don't know how cargo-careful works or how your sysroot is built but:

• if linking is needed in your tool and you want things linked like a regular nightly would, then you could e.g. copy the entire gcc-ld directory from an existing sysroot, or let bootstrap build it for you with the lld config.
• if the actual linker used is not important, you can unconditionally revert to the default linker via manual flags or rustflags, which seems easier

[RalfJung]

I will likely do the sysroot copying. Bootstrao can't help as there is no rustc checkout involved here. Is there some way to know all the files/folders that need copying? I guess for now I will just copy "bin".

[lqd]

Is there some way to know all the files/folders that need copying?

For a general sysroot I couldn't say -- especially with sanitizers, self-contained linking and their object files, etc -- but for this case copying lib/rustlib/$target/bin will indeed give you both the linker and the various target-specific wrappers that invoke it (they are in the gcc-ld subfolder). I expect that it would work fine. I imagine not all the wrappers will be useful, but they're quite small so probably not worth filtering out.