Introduce OnceVec<T> primitive and use it for AllocId caches #136105
Conversation
This significantly reduces contention when running miri under -Zthreads, allowing us to scale to 30ish cores (from ~7-8 without this patch).
rustbot
added
S-waiting-on-review
|
Some changes occurred to the CTFE / Miri interpreter cc @rust-lang/miri, @rust-lang/wg-const-eval |
|
@bors try @rust-timer queue |
This comment has been minimized.
This comment has been minimized.
rustbot
added
the
S-waiting-on-perf
Introduce OnceVec<T> primitive and use it for AllocId caches This significantly reduces contention when running miri under -Zthreads, allowing us to scale to 30ish cores (from ~7-8 without this patch). This primitive can likely replace the impls in [sync/vec.rs](https://github.com/rust-lang/rust/blob/master/compiler/rustc_data_structures/src/sync/vec.rs) AppendOnlyVec (which has a single spinlock for writes) and AppendOnlyIndexVec (rwlock) structures with better (at least) concurrent performance, too. It might also be an improvement on the current locking for Symbol interning; we should be able to make Symbol::as_str() lock-free with this structure. r? ghost (for now, until we have perf results) cc https://rust-lang.zulipchat.com/#narrow/channel/187679-t-compiler.2Fwg-parallel-rustc/topic/Miri.20not.20getting.20as.20much.20parallelism.20as.20expected
|
The job Click to see the possible cause of the failure (guessed by this bot) |
| /// This provides amortized, concurrent O(1) access to &T, expecting a densely numbered key space | ||
| /// (all value slots are allocated up to the highest key inserted). | ||
| pub struct OnceVec<T> { | ||
| // Provide storage for up to 2^35 elements, which we expect to be enough in practice -- but can |
There was a problem hiding this comment.
There can't be more than like 2^29 elements without proc macros as you only get 2^32 bytes for all source code combined and you need on average multiple characters for each identifier. In other words support for 2^35 elements is plenty.
There was a problem hiding this comment.
I think that's probably not true for e.g. AllocIds, right? At least with miri, you can presumably run a program that runs indefinitely and so allocates filling up memory here. You'd probably run out of RAM, etc., so I'm not actually particularly worried though :)
There was a problem hiding this comment.
In Miri (and CTFE), you can keep allocating and freeing memory and that way use up AllocId without using up more and mire RAM.
There was a problem hiding this comment.
... but if you do that, those allocations don't end up in alloc_map; only the final value of a const/static is put there.
|
☀️ Try build successful - checks-actions |
This comment has been minimized.
This comment has been minimized.
| ); | ||
| next | ||
| fn reserve(&self) -> AllocId { | ||
| let next_id = self.next_id.fetch_add(1, std::sync::atomic::Ordering::Relaxed); |
There was a problem hiding this comment.
| let next_id = self.next_id.fetch_add(1, std::sync::atomic::Ordering::Relaxed); | |
| let next_id = self.next_id.fetch_add(1, std::sync::atomic::Ordering::Relaxed); | |
| // Technically there is a window here where we overflow and then another thread | |
| // increments `next_id` *again* and uses it before we panic and tear down the entire session. | |
| // We consider this fine since such overflows cannot realistically occur. |
The alternative would be to use a CAS loop to avoid ever storing 0 into next_id.
| @@ -389,35 +390,32 @@ pub const CTFE_ALLOC_SALT: usize = 0; | |||
|
|
|||
| pub(crate) struct AllocMap<'tcx> { | |||
| /// Maps `AllocId`s to their corresponding allocations. | |||
| alloc_map: FxHashMap<AllocId, GlobalAlloc<'tcx>>, | |||
| alloc_map: rustc_data_structures::sync::OnceVec<GlobalAlloc<'tcx>>, | |||
There was a problem hiding this comment.
Note that this map is very sparse in practice. Most AllocId don't end up in the final value of a const/static and hence they are never put in this map. A Vec seems like a poor representation choice for that.
There was a problem hiding this comment.
Hm, so we call reserve and then don't actually insert it? I thought (I guess incorrectly) that we always did end up inserting.
If so, then it might be sufficient to just replace the reservation with a fetch_add + shard the map.
I guess we'll see how bad the max-rss regression is to help gauge.
There was a problem hiding this comment.
Or maybe even not sharding the map - if it's rare enough to actually access it, just speeding up reservation might be good enough. Could even shard the counter then.
There was a problem hiding this comment.
Hm, so we call reserve and then don't actually insert it?
Correct. We always reserve because it may be the case that we need to insert the allocation later, but most allocations are never inserted. (Well, in Miri we actually know it will never be inserted. Not sure if that is worth exploiting somehow.)
There was a problem hiding this comment.
if it's rare enough to actually access it,
Reading it is more common, but Miri only accesses the map the first time a given interpreter session accesses some const/static. At that point the allocation is copied into interpreter-local state and then the global state is never accessed again.
So yeah I think just making the counter lock-free will give the same speedup.
There was a problem hiding this comment.
Looking at density on the sysroot crates (libcore -> libtest) -- see numbers below -- I get pretty high densities in practice.
I guess those crates do not contain complicated consts with many intermediate allocations.
In a Miri run, this will be extremely sparse... or at least, at some point the counter will keep increasing but nothing new is added to the map. However, due to queries this can be interleaved I think, so if Miri first creates 10k allocations and then evaluates a const, there'll be a gap of 10k IDs in the map.
There was a problem hiding this comment.
Hm, so I tried just the lock-free counter, and it has same-ish performance to without this patch entirely (8, maybe 9 cores).
Odd, I am not sure what keeps accessing that map after the allocations have been copied into local interpreter memory.
There was a problem hiding this comment.
https://github.com/rust-lang/rust/blob/master/compiler/rustc_const_eval/src/interpret/memory.rs#L832
rust/src/tools/miri/src/borrow_tracker/mod.rs
Line 218 in 5545959
I think this is called for all root allocids on provenance GC? It's not clear to me whether that means "all" in practice for the workloads in question.
There was a problem hiding this comment.
Ah... yes that is called a lot. We should probably query the self.memory maps before querying the global map.
| // We just reserved, so should always be unique. | ||
| assert!( | ||
| self.alloc_map | ||
| .alloc_map |
There was a problem hiding this comment.
Where does the double-alloc_map come from?
|
Finished benchmarking commit (d457f92): comparison URL. Overall result: ✅ improvements - no action neededBenchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf. @bors rollup=never Instruction countThis is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.
Max RSS (memory usage)Results (primary -0.9%, secondary -4.2%)This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesResults (secondary -7.9%)This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Binary sizeThis benchmark run did not return any relevant results for this metric. Bootstrap: 773.224s -> 773.199s (-0.00%) |
rustbot
removed
the
S-waiting-on-perf
Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
…=compiler-errors interpret: is_alloc_live: check global allocs last See rust-lang#136105 (comment). (A perf run makes no sense as this is only used by Miri.)
Rollup merge of rust-lang#136166 - RalfJung:interpet-is-alloc-live, r=compiler-errors interpret: is_alloc_live: check global allocs last See rust-lang#136105 (comment). (A perf run makes no sense as this is only used by Miri.)
…-errors interpret: is_alloc_live: check global allocs last See rust-lang/rust#136105 (comment). (A perf run makes no sense as this is only used by Miri.)
|
This is superseded by #136115, I think? |
…lfJung Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
…lfJung Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
…lfJung Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang/rust#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
Shard AllocMap Lock This improves performance on many-seed parallel (-Zthreads=32) miri executions from managing to use ~8 cores to using 27-28 cores, which is about the same as what I see with the data structure proposed in rust-lang/rust#136105 - I haven't analyzed but I suspect the sharding might actually work out better if we commonly insert "densely" since sharding would split the cache lines and the OnceVec packs locks close together. Of course, we could do something similar with the bitset lock too. Either way, this seems like a very reasonable starting point that solves the problem ~equally well on what I can test locally. r? `@RalfJung`
This significantly reduces contention when running miri under -Zthreads, allowing us to scale to 30ish cores (from ~7-8 without this patch).
This primitive can likely replace the impls in sync/vec.rs AppendOnlyVec (which has a single spinlock for writes) and AppendOnlyIndexVec (rwlock) structures with better (at least) concurrent performance, too. It might also be an improvement on the current locking for Symbol interning; we should be able to make Symbol::as_str() lock-free with this structure.
r? ghost (for now, until we have perf results)
cc https://rust-lang.zulipchat.com/#narrow/channel/187679-t-compiler.2Fwg-parallel-rustc/topic/Miri.20not.20getting.20as.20much.20parallelism.20as.20expected