Re-enable atomic loads and stores for all RISC-V targets

[SimonSapin]

This roughly reverts PR #66548

Atomic "CAS" are still disabled for targets without the “A” Standard Extension for Atomic Instructions. However this extension only adds instructions for operations more complex than simple loads and stores, which are always atomic when aligned.

In the Unprivileged Spec v. 20191213 section 2.6 Load and Store Instructions of chapter 2 RV32I Base Integer Instruction Set (emphasis mine):

Even when misaligned loads and stores complete successfully, these accesses might run extremely slowly depending on the implementation (e.g., when implemented via an invisible trap). Further-more, whereas naturally aligned loads and stores are guaranteed to execute atomically, misaligned loads and stores might not, and hence require additional synchronization to ensure atomicity.

Unfortunately PR #66548 did not provide much details on the bug that motivated it, but #66240 and #85736 appear related and happen with targets that do have the A extension.

[rust-highfive]

r? @nagisa

(rust-highfive has picked a reviewer for you, use r? to override)

[rust-highfive]

⚠️ Warning ⚠️

• These commits modify compiler targets. (See the Target Tier Policy.)

[SimonSapin]

#66240 and #85736 appear related and happen with targets that do have the A extension.

I don’t know if this is relevant or what LLVM does there, but the A extension has a limitation in that single-instruction atomic swap/add/and/or/xor/min/max only exists for 32-bit or (on RV64) 64-bit values. Similar operations on 8-bit or 16-bit values would have to be a loop based on 32-bit load-reserved and store-conditional. I think this is the kind of loop AtomicU8::fetch_update does?

[MabezDev]

I don't think changing the max_atomic_width is enough. The LLVM backend currently will expand load/stores to libcalls on targets without the +a extension. I'm not sure what codegen granularity we have in rustc, but it would require us to do some lowering ourselves before passing to LLVM to make sure there are appropriate fences in place. We would also need to guarentee this custom lowering never happens on targets with atomic_cas: true, just incase the CAS routines use a locking data structure to implement CAS.

More details in here: #81752 (comment).

[jamesmunns]

As an analogy, this should be the same status as armv5te and thumbv6m targets, where they do not naturally have any ability to perform CAS operations, but do have atomic load/stores of their native word width.

It seems as though this may be handled differently upstream in LLVM for RV32I targets, though that seems like something that should be addressed upstream for the sake of consistency.

[SimonSapin]

Trying this on a somewhat(?) simple project fails:

error: linking with `rust-lld` failed: exit status: 1
  |
  = note: "rust-lld" "-flavor" "gnu" "/tmp/rustcZzSyCr/symbols.o" […]
  = note: rust-lld: error: undefined symbol: __atomic_load_4
          >>> referenced by compiler_builtins.5931568b-cgu.83
          >>>               compiler_builtins-35b1260b00e1afde.compiler_builtins.5931568b-cgu.83.rcgu.o:(compiler_builtins::mem::memcpy::h4f55b8ec9004b8fa) in archive /home/simon/projects/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/riscv32i-unknown-none-elf/lib/libcompiler_builtins-35b1260b00e1afde.rlib

Which looks like #85736

[MabezDev]

I think that is the intended behaviour, even though we're setting the max_atomic_width, LLVM still won't generate the code because it knows the target doesn't have the A extension, instead, it defers to libcalls.

Along with this PR, we need to convince upstream LLVM that adding atomic load/stores for RISCVI targets is safe. The only way we can do that is if we can convey to LLVM that mixing true atomic load/stores with locking atomic CAS implementations is not possible. I think this is only possible under two conditions:

atomic_cas is set to false in the target definition

or

the cas implementation is known to be lock-free

The second seems quite difficult to prove, but the first one is easy. We just need a way of exposing this information to LLVM.

[MabezDev]

I suppose another option is to implement the libcalls in software, inside https://github.com/rust-lang/compiler-builtins, but that would require convincing them to change their current stance which is

Rust only exposes atomic types on platforms that support them and therefore does not need to fall back to software implementations.

We would also have to abide by the rules stated above (not mixing real atomics with locking atomic implementations) but perhaps this is easier to verify on the Rust side?

@Amanieu any thoughts on this?

[SimonSapin]

The fence instruction is part of the base RV32I ISA, and core::sync::atomic::fence seems to be available on all targets.

As a work around, this generates on both riscv32i-unknown-none-elf and riscv32imac-unknown-none-elf the same machine code as AtomicU8 does on riscv32imac-unknown-none-elf:

pub struct MyAtomicU8(UnsafeCell<u8>);

unsafe impl Sync for MyAtomicU8 {}

impl MyAtomicU8 {
    pub fn store(&self, value: u8, ordering: Ordering) {
        fence(ordering);
        unsafe { self.0.get().write(value) }
    }

    pub fn load(&self, ordering: Ordering) -> u8 {
        let value = unsafe { self.0.get().read() };
        fence(ordering);
        value
    }
}

Does it look correct?

[taiki-e]

As a work around, this generates on both riscv32i-unknown-none-elf and riscv32imac-unknown-none-elf the same machine code as AtomicU8 does on riscv32imac-unknown-none-elf:

Does it look correct?

It does not look correct:

• ptr::{write,read} are not atomic operations. core::ptr docs says:

  All accesses performed by functions in this module are non-atomic in the sense of atomic operations used to synchronize between threads. This means it is undefined behavior to perform two concurrent accesses to the same location from different threads unless both accesses only read from memory. Notice that this explicitly includes read_volatile and write_volatile: Volatile accesses cannot be used for inter-thread synchronization.

• IIUC, atomic fences are guaranteed to work only for atomic operations. The compiler can probably reorder non-atomic operations. EDIT: this does not seem correct

AFAIK, the only sound way with pure Rust currently available is to use inline assembly like my portable-atomic crate does.

(In this particular case, volatile read/write + fence probably work too, but is also not sound because volatile read/write are not guaranteed to be atomic. Actualy, it seems LLVM uses instructions that are not guaranteed to be atomic in aarch64's volatile read/write.)

[SimonSapin]

This is very helpful, thanks @taiki-e!

[Amanieu]

Overall I'm in favor of this change, but it requires either one of the following to happen:

• LLVM stops emitting libcalls for RISC-V atomic load/store.
• We implement __atomic_load/__atomic_store in compiler-builtins.

I'm happy with either approach.

• IIUC, atomic fences are guaranteed to work only for atomic operations. The compiler can probably reorder non-atomic operations. core::sync::atomic::fence docs says:

No, atomic fences also enforce ordering on non-atomic operations.

[MabezDev]

Thanks for weighing in @Amanieu! The LLVM changes are a bit out of my depth, but I could PR some __atomic_load/__atomic_store implementations for RISCV in compiler-builtins?

To expand on my earlier comment about mixing lock-free load/stores with locking cas implementations, see this LLVM discussion thread: https://reviews.llvm.org/D47553. How do you think we should avoid that? Only provide libcall implementations if atomic_cas: false? Is there a compiler cfg for this? Maybe not worrying about this at all is an option, thats what GCC does...

[SimonSapin]

Looks like libcore uses cfg(target_has_atomic = "8") as "has CAS atomics for size 8". (cfg(target_has_atomic_load_store = "8") is what makes the Atomic* types exist at all.)

[Amanieu]

To expand on my earlier comment about mixing lock-free load/stores with locking cas implementations, see this LLVM discussion thread: reviews.llvm.org/D47553. How do you think we should avoid that? Only provide libcall implementations if atomic_cas: false? Is there a compiler cfg for this? Maybe not worrying about this at all is an option, thats what GCC does...

The fundamental issue is summary in the last comment here: this is unsound when mixing locked-based CAS operations with lock-free load/store operations. While this doesn't apply to Rust since we don't expose CAS operations on this target, we still shouldn't override the __atomic_* symbols since those may be used by C code.

A better solution might be to explicitly lower atomic load/store to a volatile load/store + a fence in rustc or in the standard library.

[nagisa]

r? @Amanieu

[Amanieu]

LLVM seems to be moving away from supporting atomic load/store independently of full atomic support. We need to make a decision of what to support in Rust: #99595 (comment)

[jamesmunns]

LLVM seems to be moving away from supporting atomic load/store independently of full atomic support. We need to make a decision of what to support in Rust: #99595 (comment)

Hey @Amanieu, what would be the best way to open up a meta issue and get the embedded-wg involved?

It seems like this is a change to llvm, and it might impact many embedded targets, potentially in a breaking way if atomics with just load/stores are removed for stable targets.

[bors]

📌 Commit 20bd0c3 has been approved by Amanieu

It is now in the queue for this repository.

[bors]

⌛ Testing commit 20bd0c3 with merge 90f0b24...

[bors]

☀️ Test successful - checks-actions
Approved by: Amanieu
Pushing 90f0b24 to master...

[taiki-e]

@Amanieu I believe we need to also enable forced-atomics target feature for these targets (otherwise, libcalls are generated): https://godbolt.org/z/433qeG7vd

And, forced-atomics target feature itself is currently broken due to another bug: #114153

[rust-timer]

Finished benchmarking commit (90f0b24): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.4%, 1.3%]	18
Regressions ❌ (secondary)	0.6%	[0.2%, 0.9%]	12
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[0.4%, 1.3%]	18

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.5%	[-1.8%, -1.2%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-1.5%	[-1.8%, -1.2%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 650.501s -> 650.303s (-0.03%)

[lqd]

While this PR may be reverted, the performance results are noise, since it only modified RISC-V target specs, so: @rustbot label: +perf-regression-triaged

[taiki-e]

Ok, I can confirm this PR break the build for these targets with "undefined symbol" error.

• https://github.com/taiki-e/atomic-maybe-uninit/actions/runs/5773850762/job/15650274924
• https://github.com/taiki-e/semihosting/actions/runs/5773864026/job/15650258088

[Mark-Simulacrum]

While this PR may be reverted, the performance results are noise, since it only modified RISC-V target specs, so:
@rustbot label: +perf-regression-triaged

Agreed with changing the label, just also wanted to reference #114481 -- my sense is the noise here is quite similar to that PRs.