Make #[used] work when linking with ld64

[madsmtm]

To make #[used] work in static libraries, we use the symbols.o trick introduced in #95604.

However, the linker shipped with Xcode, ld64, works a bit differently from other linkers; in particular, it completely ignores undefined symbols by themselves, and only consider them if they have relocations (something something atoms something fixups, I don't know the details).

So to make the symbols.o file work on ld64, we need to actually insert a relocation. That's kinda cumbersome to do though, since the relocation must be valid, and hence must point to a valid piece of machine code, and is hence very architecture-specific.

Fixes #133491, see that for investigation.

---

Another option would be to pass -u _foo to the final linker invocation. This has the problem that -u causes the linker to not be able to dead-strip the symbol, which is undesirable. (If we did this, we would possibly also want to do it by putting the arguments in a file by itself, and passing that file via @, e.g. @undefined_symbols.txt, similar to #52699, though that is only supported since Xcode 12, and I'm not sure we wanna bump that).

Various other options that are probably all undesirable as they affect link time performance:

• Pass -all_load to the linker.
• Pass -ObjC to the linker (the Objective-C support in the linker has different code paths that load more of the binary), and instrument the binaries that contain #[used] symbols.
• Pass -force_load to libraries that contain #[used] symbols.

Failed attempt: Embed -u _foo in the object file with LC_LINKER_OPTION, akin to #121293. Doesn't work, both because ld64 doesn't read that from archive members unless it already has a reason to load the member (which is what this PR is trying to make it do), and because ld64 only support the -l, -needed-l, -framework and -needed_framework flags in there.

---

TODO:

• Support all Apple architectures.
• Ensure that this works regardless of the actual type of the symbol.
• Write up more docs.
• Wire up a few proper tests.

@rustbot label O-apple

[rustbot]

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[madsmtm]

I'm quite unhappy with this, it feels very brittle to emit a manual relocation like this, but I think it's the only way to do it (without pulling in a bunch of LLVM codegen internals).

I'm also unsure of the correctness, there might be more target options and/or flags that we should be checking here? Or maybe we should emit different kinds of relocations based on the kind of symbol?

[thomcc]

Yeah... emitting manual relocations like this feels super fragile and hard to maintain. At the very least this would need comments and ideally references.

[DianQK]

I don't think additional comments are needed here, except to explain what this data represents, such as [0, 0, 0, 0x14] represents the b instruction.

[DianQK]

I have a bit concern about here, the symbol of #[used] is a global variable, the C code should be void foo(void);. I'm not sure what can happen here. I prefer an address offset instruction. This is more safe to me.

[DianQK]

You may want to add a local symbol to help ld64 strip these dead code.

$ objdump -d target/release/divan-example | head -n 10

target/release/divan-example:   file format mach-o arm64

Disassembly of section __TEXT,__text:

000000010000273c <__text>:
10000273c: 1400000a     b       0x100002764 <_main>
100002740: 14016d1e     b       0x10005dbb8 <__ZN3std9panicking11EMPTY_PANIC17hd7edc6c0e88a95c9E>
100002744: 1400eca4     b       0x10003d9d4 <_rust_eh_personality>
100002748: 14016632     b       0x10005c010 <__ZN13divan_example4PUSH17h56090c2dad718183E>

[madsmtm]

Your suggestions here sounds correct to me, I've also later been thinking that an address offset instruction would be safer than a jump (since address offsets can be done safely on both function pointers and data pointers).

You may want to add a local symbol

I'm not sure I understand what you mean by that? You want a local symbol that appears as the "entry" point for the section? Does that help the linker see that the code is unused?

[madsmtm]

Note that we could outsource part of this to object by using object::write::RelocationFlags::Generic, but we'd still have to write the machine code ourselves.

[madsmtm]

@estebank I'm gonna mark this as ready for an initial review, to make sure that you agree that the approach here is the right one (vs. the alternatives I noted in the PR description / another alternative that I haven't considered). If so, then I'll complete the remaining TODO items.

[rustbot]

This PR modifies tests/run-make/. If this PR is trying to port a Makefile
run-make test to use rmake.rs, please update the
run-make port tracking issue
so we can track our progress. You can either modify the tracking issue
directly, or you can comment on the tracking issue and link this PR.

cc @jieyouxu

[estebank]

I'm sorry. I don't think I'm the best person to review this change.

r? compiler

[jieyouxu]

I'm going to ask in T-compiler since this is likely to bounce based on random rerolls. I'll give it a few days, if we still can't find a suitable reviewer by then, I'll compiler-nominate this PR.

Please ping me next Monday if we still haven't found a suitable review (in case this gets lost in my review queue), I'll nominate it for next week's triage meeting.

[DianQK]

Edit: I need more time to check it. I'm not the best person here, but fewer people are using macOS here.

I will try to review it tomorrow.
r? DianQK

[DianQK]

IIUC, I can reproduce this with C:

// foo.c
__attribute__((constructor))
void foo() {}

// main.c
int main() {
    return 0;
}

When I build the foo.c to the libfoo.a, foo.o that in libfoo.a never be loaded by any linker. For fix this, we added a symbols.o to help the linker load such object files.

I'm concerned about the current implementation. It seems we're building our logic around ld64's internal implementation details, and ld64 isn't truly open source. (I haven't looked into ld64's implementation details yet. I'm not sure if this approach is fragile or will be difficult to maintain in the future.)

IIUC, the key here is to have the linker load object files as intended, just like when we directly use ld foo.o. So why don't we just add these object files directly?

If we don't want to do this, how about adding the right value of #[used] to symbols.o? For static PUSH: extern "C" fn() = push;, we add "push" to symbols.o. That looks workable, as long as push always be a global symbol and both PUSH and push stay in the same object file.

[DianQK]

This link doesn't seem to mention the details of how ld64 resolves undefined symbols, especially regarding relocations. Should we add links to the relevant code locations?

[madsmtm]

Yeah, it was the closest I could find, but actual links to the code would be a good idea. I'll try to find the relevant parts when I get the time.

Note though that I think the new linker, -ld_prime, is not open source? So this will only be a weak reference in any case.

[madsmtm]

[...]
When I build the foo.c to the libfoo.a, foo.o that in libfoo.a never be loaded by any linker. For fix this, we added a symbols.o to help the linker load such object files.

That's correct.

I'm concerned about the current implementation. It seems we're building our logic around ld64's internal implementation details, and ld64 isn't truly open source. (I haven't looked into ld64's implementation details yet. I'm not sure if this approach is fragile or will be difficult to maintain in the future.)

I agree that it is fragile, but I don't think it is as bad as it looks; consider that linkers are mostly backwards compatible, and that it works now (whereas before it just didn't work at all, in no versions of ld64). I honestly don't think Apple is gonna change how this works, and if they do, we'll have the Xcode betas to fix it.

The only thing I can conceivably think of that might break is:

• If symbols were treated differently depending on their type (function vs. static).
• If the linker figures out that the data that the relocation refers to is never actually used.

I guess if we wanted to make it even more robust, we'd do something like:

// symbols.rs
extern crate crate1;
extern crate crate2;
extern crate crate3;

fn _used_symbols() {
    let _static1 = crate1::STATIC1;
    let _static2 = crate2::STATIC2;
    let _fn3 = crate3::fn3;
}

// rustc symbols.rs --emit=obj

That is, emit a label that refers to the block of code that touches all symbols, such that the linker cannot assume the section to be unused. The roughly equivalent could be done with object-rs like so:

file.add_symbol(write::Symbol {
    name: "_used_symbols".into(),
    value: 0,
    size: 0,
    kind: SymbolKind::Text,
    scope: SymbolScope::Dynamic,
    weak: false,
    section: write::SymbolSection::Section(section_id),
    flags: SymbolFlags::None,
});

Was that clear? I can try to go in more detail here if you want? Or try to conjure up some contrived assembly code, and consider how ld64 would have to link that now and in the future?

IIUC, the key here is to have the linker load object files as intended, just like when we directly use ld foo.o. So why don't we just add these object files directly?

I can see two ways to do that:

• Avoid archives altogether, just pass each object file in every crate to the linker (or maybe combine the different object files generated by each codegen unit into a "libfoo.o").
• Link just the object files that use #[used]. Requires somehow extracting the relevant object files from the archive at link time, or maybe doing it while building the crate itself? E.g. create libfoo.a and foo_used_symbols.o.

The former is bad for link-time performance (the linker can skip a lot of work for archives that if can't for object files), and the latter is either also bad for perf, or makes the integration between rustc and Cargo even more complex than it already is.

If we don't want to do this, how about adding the right value of #[used] to symbols.o? For static PUSH: extern "C" fn() = push;, we add "push" to symbols.o. That looks workable, as long as push always be a global symbol and both PUSH and push stay in the same object file.

Hmm, not sure I understand?

Note that we'd still want e.g. #[used] #[link_section = "..."] static FOO: i32 = 2; to be seen by the linker, and here there's no "inner" symbol to refer to?

[bjorn3]

If the linker figures out that the data that the relocation refers to is never actually used.

The linker would have still pulled in the object file by that time. And #[used] doesn't actually have to prevent linker GC of the section. Only make sure that it isn't discarded before the linker has a chance of see the symbol and make it's own decision about whether it is considered to be a root for the linker GC. The unstable #[used(linker)] also forces the linker to keep the symbol, but that emits a separate flag on the symbol to prevent it from being deleted by the linker when the linker actually sees it.

[DianQK]

Note that we'd still want e.g. #[used] #[link_section = "..."] static FOO: i32 = 2; to be seen by the linker, and here there's no "inner" symbol to refer to?

Ah, yeah! This also reminder me of something. Consider the following C code:

__attribute__((constructor)) static void push() {}

or

static void push() {}

#if defined(__linux__)
__attribute__((section(".init_array")))
#elif defined(__APPLE__)
__attribute__((section("__DATA,__mod_init_func,mod_init_funcs")))
#endif
static void (*const PUSH)(void) = push;

There are no global symbols here. I think rustc should not modify local symbols into global symbols through #[used] - in this case, it's the same situation as the C code above, where an object file has no global symbols.

IMO, loading object files through a "symbols.o" doesn't seem like a good idea. Looking at the longer term, I don't think we should continue to go this way.

Looking at this PR itself, I don't think we need to focus on ld64's internal implementation, since we're trying to simulate a standard object file. Based on my newly added comments, I think this approach is still quite complicated, and we need more workarounds to resolve it. What do you think?

[DianQK]

I don't think additional comments are needed here, except to explain what this data represents, such as [0, 0, 0, 0x14] represents the b instruction.

[DianQK]

I have a bit concern about here, the symbol of #[used] is a global variable, the C code should be void foo(void);. I'm not sure what can happen here. I prefer an address offset instruction. This is more safe to me.

[DianQK]

You may want to add a local symbol to help ld64 strip these dead code.

$ objdump -d target/release/divan-example | head -n 10

target/release/divan-example:   file format mach-o arm64

Disassembly of section __TEXT,__text:

000000010000273c <__text>:
10000273c: 1400000a     b       0x100002764 <_main>
100002740: 14016d1e     b       0x10005dbb8 <__ZN3std9panicking11EMPTY_PANIC17hd7edc6c0e88a95c9E>
100002744: 1400eca4     b       0x10003d9d4 <_rust_eh_personality>
100002748: 14016632     b       0x10005c010 <__ZN13divan_example4PUSH17h56090c2dad718183E>

[madsmtm]

There are no global symbols here. I think rustc should not modify local symbols into global symbols through #[used] - in this case, it's the same situation as the C code above, where an object file has no global symbols.

Hmm, I think I see what you mean, but is it really an issue? My understanding is vague here, but I think global vs. local symbols are linker-only concepts, are they not? Perhaps things differ when it comes to dynamic libraries, but I don't the symbol is marked as "exported" (?) , so the linker should not treat things any differently there either?

IMO, loading object files through a "symbols.o" doesn't seem like a good idea. Looking at the longer term, I don't think we should continue to go this way.

I'm not sure I understand, how do you propose we solve this instead, then? In a way, #[used] has to make the symbol global for it to be seen by the linker, right? That's what you want when using #[used].

[bjorn3]

(technically #[used] is equal to #[used(compiler)] only means the compiler will keep the function, not the linker will keep the function. the latter needs the unstable #[used(linker)]. That said, #[used] meaning #[used(compiler)] was a mistake IMO and we probably should make it #[used(linker)] in the future.)

[DianQK]

There are no global symbols here. I think rustc should not modify local symbols into global symbols through #[used] - in this case, it's the same situation as the C code above, where an object file has no global symbols.

Hmm, I think I see what you mean, but is it really an issue? My understanding is vague here, but I think global vs. local symbols are linker-only concepts, are they not? Perhaps things differ when it comes to dynamic libraries, but I don't the symbol is marked as "exported" (?) , so the linker should not treat things any differently there either?

IMO, loading object files through a "symbols.o" doesn't seem like a good idea. Looking at the longer term, I don't think we should continue to go this way.

In a way, #[used] has to make the symbol global for it to be seen by the linker, right? That's what you want when using #[used].

As @bjorn3 described, #[used] only indicates that the symbol should be kept, rather than making it as a global symbol. Let's look at an example where we don't need #[used]. Note that both FOO and foo are local symbols here, and the object file cannot be loaded by an undefined symbol. The local and global symbols here describe the visibility of symbols within the object file.

$ cat main.rs
#![allow(dead_code)]

#[cfg_attr(target_os = "linux", link_section = ".init_array")]
#[cfg_attr(
    target_os = "macos",
    link_section = "__DATA,__mod_init_func,mod_init_funcs"
)]
static FOO: fn() = foo;
fn foo() {
    println!("foo");
}

fn main() {}

$ rustc main.rs && ./main
foo

$ nm main | grep -i foo
0000000100044010 s __ZN4main3FOO17h1b58eb28acf3c7f7E
0000000100000cf8 t __ZN4main3foo17h86e604b79f2f44edE

I'm not sure I understand, how do you propose we solve this instead, then?

My current thinking is to treat this PR as a working solution for now, while loading direct object file is a long-term improvement goal. Besides the test case, could you address my two concerns above?

1. Ensure that the instructions we added in symbols.o can be eliminated
2. Make these instructions potentially safer by using address offset instruction

[bjorn3]

As @bjorn3 described, #[used] only indicates that the symbol should be kept, rather than making it as a global symbol.

Not even kept by the linker. Just kept long enough for the linker to be able to see it at all. The linker is free to discard the symbol if it wants to. It just has to see it such that it can make the decision whether to discard it or not. This makes it pretty much useless for any purpose other than global constructors. Even the section merging for distributed slices is broken with lld because of this and requires #[used(linker)] instead AFAIK.

[madsmtm]

My current thinking is to treat this PR as a working solution for now, while loading direct object file is a long-term improvement goal.

Ah, cool! Thanks for the review and discussion so far, btw!

Besides the test case, could you address my two concerns above?

Certainly (though it might take a little while for me to get back into it)!
@rustbot author

This makes it pretty much useless for any purpose other than global constructors.

I get your point that it might be pretty much useless on its own, but you can very much make it useful by adding no_dead_strip (or similar for other platforms) to the linker section.

[bjorn3]

but you can very much make it useful by adding no_dead_strip (or similar for other platforms) to the linker section.

Adding such a flag to the symbol is what the unstable #[used(linker)] does. Outside of Mach-O rustc doesn't expose any way to set it for the entire section. And ELF doesn't have a way to set it for the entire section as opposed to individual symbols afaik.