native_libs: More consistent native library search between rustc and linker

[petrochenkov]

Continuation of #129366.

Typically rustc will delegate native library search to linker, by passing to it library search directory options, like -L or equivalents.

In some cases, however, rustc will search for native libraries "manually", by itself.

1. When bundling libraries into rlibs and staticlibs
2. When doing search instead of the linker, when linker simply doesn't support search by name
   • Example: -force_load with ld64, which is required to implement +whole-archive, and only accepts full paths instead of names
3. When supporting libraries with alternative naming scheme not supported by linker
   • Example: searching for libfoo.a on MSVC (linker: Allow MSVC to use import libraries following the Meson/MinGW convention #123436)

It is important for the rustc search and for the linker search to give the same result (modulo cases 1. and 3., where the difference is intentional).
It is not always possible, because linkers may also search for some default system directories that depend on a specific system and are hard to predict in general.
So we can make a weaker requirement (*) - the directory list searched by rustc must be a prefix of the directory list considered by linker.

In this PR I try to use the full list of directories passed as -Ls to linker for rustc's search as well.
Besides directories explicitly provided by user, it also includes sysroot, self-contained sysroot and some SDK directories on Apple targets.

This is not a full solution, because the library bundling logic in rustc_metadata (as opposed to rustc_codegen_ssa) still skips self-contained sysroot and the Apple SDK directories.
Determining self_contained_components and apple_sdk_root requires a large chunk of linker-probing logic to be available in rustc_metadata, which generally doesn't do any linking at all.
On the other hand, turning get_self_contained_components() and get_apple_sdk_root() into queries to avoid reevaluating them from different places is also pretty annoying, because the logic in rustc_codegen_ssa cannot use queries, so they would need to be evaluated earlier and the results would need to be stored in CodegenResults.

I'm not yet sure this is the right direction, maybe we should instead exclude all kinds of sysroots from the rustc's "manual" native library search. That would satisfy the requirement (*) as well.
r? @bjorn3

[rustbot]

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

[petrochenkov]

maybe we should instead exclude all kinds of sysroots from the rustc's "manual" native library search

I'll post this alternative as well, and do a couple of other experiments, before considering this PR for merging.
@rustbot author

[bors]

☔ The latest upstream changes (presumably #138202) made this pull request unmergeable. Please resolve the merge conflicts.

[petrochenkov]

Determining self_contained_components and apple_sdk_root requires a large chunk of linker-probing logic to be available in rustc_metadata, which generally doesn't do any linking at all.

Moving the information about bundled libraries from regular metadata (where it's not necessary) to some kind of rlib digest (#138243) will help to avoid doing any kind of library lookup in rustc_metadata and do it all in rustc_codegen_ssa.

[petrochenkov]

Closing this in favor of #138273 for now, but it can be reconsidered later if #138243 is implemented.