rust-lld since 1.38 overlaps .text with .rodata for embedded arm target

[lightblu]

Minimal blinky example for a cortex-m4 using cortex-m-rt crate (and a svd2rust generated device crate) since 1.38 (also present with 1.40 nightly) fails linking:

error: linking with `rust-lld` failed: exit code: 1
  |
  = note: "rust-lld" "-flavor" "gnu" "-L" "/home/.rustup/toolchains/    nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabihf/lib" "/home/work/remb    /target/thumbv7em-none-eabihf/debug/deps/remb-0e89ac955194710a.1u2vah3l27soigvo.rcgu.o" "/    home/lb/work/remb/target/thumbv7em-none-eabihf/debug/deps/    remb-0e89ac955194710a.20vhaom7k5jk5m2z.rcgu.o" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/remb-0e89ac955194710a.21jk4vxquq5fsklp.rcgu.o" "/home/    work/remb/target/thumbv7em-none-eabihf/debug/deps/    remb-0e89ac955194710a.2v63og99w4q7umup.rcgu.o" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/remb-0e89ac955194710a.40ihsdzg5hvegvya.rcgu.o" "/home/    work/remb/target/thumbv7em-none-eabihf/debug/deps/    remb-0e89ac955194710a.4edrqqhvp54adqvm.rcgu.o" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/remb-0e89ac955194710a.ela5gwo3oi0mbal.rcgu.o" "-o" "/home/    lb/work/remb/target/thumbv7em-none-eabihf/debug/deps/remb-0e89ac955194710a"     "--gc-sections" "-L" "/home/work/remb/target/thumbv7em-none-eabihf/debug/deps" "-L" "/    home/lb/work/remb/target/debug/deps" "-L" "/home/work/remb/target/thumbv7em-none-eabihf/    debug/build/remb-4f277a380d027b54/out" "-L" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/build/cortex-m-cc9ea6fae48cdb10/out" "-L" "/home/work/remb/    target/thumbv7em-none-eabihf/debug/build/cortex-m-rt-e72a5142fcf97ff7/out" "-L" "/home/    work/remb/target/thumbv7em-none-eabihf/debug/build/cortex-m-semihosting-62a07b6b9f966603/    out" "-L" "/home/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/    thumbv7em-none-eabihf/lib" "-Bstatic" "/home/work/remb/target/thumbv7em-none-eabihf/    debug/deps/libcortex_m_rt-d442f13f5d2d3c17.rlib" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/libr0-44368e36351c6aca.rlib" "--start-group" "/home/    work/remb/target/thumbv7em-none-eabihf/debug/deps/libpanic_itm-54548bdc2864396e.rlib" "/    home/lb/work/remb/target/thumbv7em-none-eabihf/debug/deps/    libcortex_m-7bfd5c3ea6d14260.rlib" "/home/work/remb/target/thumbv7em-none-eabihf/debug/    deps/libvolatile_register-5316b243390db931.rlib" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/libvcell-8547e68540e6b860.rlib" "/home/work/remb/target    /thumbv7em-none-eabihf/debug/deps/libbare_metal-5adef261ba9f8cbc.rlib" "/home/work/remb/    target/thumbv7em-none-eabihf/debug/deps/libaligned-074dd0e0ef261e42.rlib" "/home/work/    remb/target/thumbv7em-none-eabihf/debug/deps/libas_slice-e5decefefa2360b9.rlib" "/home/    work/remb/target/thumbv7em-none-eabihf/debug/deps/    libstable_deref_trait-8119af54d84c3dce.rlib" "/home/work/remb/target/    thumbv7em-none-eabihf/debug/deps/libgeneric_array-fd3cfec1f6f858d0.rlib" "/home/work/    remb/target/thumbv7em-none-eabihf/debug/deps/libtypenum-4c848b432b16c46c.rlib" "/home/    lb/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabihf/    lib/librustc_std_workspace_core-bd695e8dc7ecd4c4.rlib" "/home/.rustup/toolchains/    nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabihf/lib/    libcore-d0e985a669afa02b.rlib" "--end-group" "/home/.rustup/toolchains/    nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7em-none-eabihf/lib/    libcompiler_builtins-58bdfb0b01f66c08.rlib" "-Tlink.x" "-Map=./Application.map" "-Bdynamic"
  = note: rust-lld: error: section .text file range overlaps with .rodata
          >>> .text range is [0x1410, 0x4471]
          >>> .rodata range is [0x41D0, 0x4B5F]
          
          rust-lld: error: section .text virtual address range overlaps with .rodata
          >>> .text range is [0x410, 0x3471]
          >>> .rodata range is [0x31D0, 0x3B5F]
          
          rust-lld: error: section .text load address range overlaps with .rodata
          >>> .text range is [0x410, 0x3471]
          >>> .rodata range is [0x31D0, 0x3B5F] 

My device needs .text to start at 0x410, for what I use the commented option in memory.x by cortex-m-quickstart generated example:

/* You can use this symbol to customize the location of the .text section */
/* If omitted the .text section will be placed right after the .vector_tablesection */
/* This is required only on microcontrollers that store some configuration right after the vector table */
_stext = ORIGIN(FLASH) + 0x410;

Without this offset, it links fine but places .text already at 016c (my device crate replaces __INTERRUPTS with 75 vectors, which makes the hole vector section fill til 0x16c only), but for the device it needs to start at 0x410.

This builds fine with rust-lld 1.37 and also with binutils ld ("-C", "linker=arm-none-eabi-ld" in .cargo/config) under 1.38 and nightly, just with rust-lld it is failing since 1.38.

Interesting is, that when I don't specify the offset in memory.x, and .text starts at 0x16c, .rodata starts at 0x31D0, too. So it seems that this offset via _stext is taken into account for .text but not for .rodata? Though I cannot see why this should happen via cortex-m-rt link script:

https://github.com/rust-embedded/cortex-m-rt/blob/master/link.x.in

I searched for relates issues but found only some which were related to alignment issues and small overlaps of some bytes; this overlap of 0x3471 - 0x31D0 = 673 bytes seems to happen because the offset of 0x410 - 0x16c = 676 bytes which puts .text fine at its correct starting position 0x410 is not obeyed for following .rodata section?

Attaching minimal example files; device.x and the vector table definition in main.rs are coming from the device crate and just have been incorporated to provide a smaller example.

minimal-rust-lld-issue.zip

[mati865]

cc @nikic

[nikomatsakis]

cc @japaric @therealprof @jamesmunns -- maybe somebody from the embedded wg has some insight into what is happening here?

[jonas-schievink]

I know that overriding _stext was somewhat broken and was fixed for the RISC-V runtime here: rust-embedded/riscv-rt#36. I don't know if those issues were also in the original cortex-m-rt, or if they were ever fixed there.

Perhaps the linker script in cortex-m-rt had other issues that have now surfaced with the newer LLD?

[jonas-schievink]

(related: would it be helpful to add a WG-embedded label for issues impacting embedded work?)

[jamesmunns]

Hey @lightblu, thanks for reporting. It seems that the behavior of LLD changed between 1.37 and 1.38 in a way that breaks the ABI of the cortex-m-rt crate (likely due to an LLVM version upgrade).

This is likely either:

• An intended change to LLVM/LLD behavior
• A regression in LLD
  • Possibly related to _stext fails to move .text with lld rust-embedded/cortex-m-rt#188 and https://reviews.llvm.org/D60744 ?

If it is the former, unfortunately as far as I understand, we have not gotten official guidance from the upstream rust project that use of custom linker scripts should be considered part of Rust's stability guarantees, so this breakage will likely need to be followed up with in the cortex-m-rt repo. For now, I will assume this is the same root cause as rust-embedded/cortex-m-rt#188, but if not we can open a separate issue.

@yodaldevoid did you ever follow up with this re: the LLVM regression in the ELF headers? Would you know how to check whether the minimal example posted above matches the same regression?

[jamesmunns]

@nikomatsakis were there any LLVM changes that landed in nightly before 2019-04-13 or so, that stabilized in 1.38 (2019-09-26 or so)? If so, that might help us diagnose the specific changes that cause this regression for stable embedded targets.

[therealprof]

It seems to me as if the newer versions of rust-ld messed up the upkeep of the internal offset tracking as soon as a custom offset is supplied. The PR referenced above adds some additional labels to track the end of the sections so we can reference them at the beginning of the next section.

I tested the minimal example provided above with both 1.34 and 1.38, before and after the PR applied. For 1.34 the linker map is identical (sans the additional labels) and with 1.38 it compiles now with only a few changes related to changes in the generated binaries.

[yodaldevoid]

I will test this minimal example later today.
EDIT: Never mind, work is killing me again this week. I'll get to this on Saturday.

A note on where things are at with rust-embedded/cortex-m-rt#188: I've taken a few passes at fixing the ELF header overwriting problem, but I haven't been able to find a solution that doesn't break some other feature of LLD. Then again, the LLD tests are not the greatest at defining what is being tested vs. what is simply due to the current implementation. I'm currently taking a bit of a break on that problem to throw myself at const-generics for a little bit.

[pnkfelix]

triage: P-high (unless someone in WG-embedded says it should not be so high priority), removing I-nominated

[therealprof]

@lightblu Did you have a chance to check out whether therealprof/cortex-m-rt@d5f2a94 is sufficient to adress your problem? We're looking to release a new version to paper over^W^W work around the issue.

CC @korken89

[yodaldevoid]

I have tested the minimal example with the LLD produced by LLVM commit ecc5e80 (it was and old master that I had checked out), and found the example to compile properly. I then checked out the latest master (commit 174967f), and found that to also compile the minimal example as well.

This all reminds me that the "Overwriting ELF sections" problem comes down to the out of order memory sections that I was using in rust-embedded/cortex-m-rt#188, so the minimal example in this issue should be no problem.

All this to say, this seems to be a regression in LLD that might have been fixed further upstream.

[pnkfelix]

Hey @lightblu , can you confirm whether the comments above have addressed the issue?

(unless I see evidence that there's something else for T-compiler to do here, I will probably close this in a week...)

[ashtneoi]

If I'm reading this issue right, it sounds like it's an lld bug. Is there an upstream bug report?

(If it's worth anything, I think I have an even more minimal RISC-V reproduction --- seems to fail even on rust master c0e02ad. I know RISC-V is tier 2 but I assume this issue is target-agnostic.)

[pnkfelix]

triage: visiting as part of general attempt to burn down set of unassigned P-high non-ICE issues. I note that I "threatened" to close this issue as "nothing further for T-compiler to do here" back in November of 2019, but it sounds ike recent reports say that there may be evidence that this is target agnostic.

Nominating for discussion at T-compiler meeting, mostly to see if anyone there has ready ability to reproduce locally.

[ashtneoi]

My RISC-V repro (this branch) still works on the latest Rust nightly, and it works with a couple changes on x86 too (this branch). I've been meaning to try it without cargo on upstream LLVM too, but I haven't gotten around to that yet.

[pnkfelix]

discussed at T-compiler meeting. Self-assigning to look at @ashtneoi 's x86 repro, and see if I can isolate if this an upstream lld bug or not.

[pnkfelix]

Visiting for P-high review

I'm not sure I ever made progress on my attempt to recreate the bug (and seeing if it was an upstream lld bug)

@lightblu can you confirm that this is still a problem for you?

[lightblu]

Sorry for never coming back here, but this is gone for some time now and hard to bring back quickly now: so no, cannot confirm anymore this is still a problem.

[jieyouxu]

P-high pre-triage: given that issue reporter cannot confirm this is still a problem, closing for now as it might have been addressed in lld. Please reopen if this is still an issue.