Revamp caching scheme in PoolingAsyncValueTaskMethodBuilder #55955
Conversation
stephentoub
added
area-System.Threading.Tasks
tenet-performance
|
Tagging subscribers to this area: @dotnet/area-system-threading-tasks Issue DetailsThe current scheme caches one instance per thread in a ThreadStatic, and then has a locked stack that all threads contend on; then to avoid blocking a thread while accessing the cache, locking is done with TryEnter rather than Enter, simply skipping the cache if there is any contention. The locked stack is capped by default at ProcessorCount*4 objects. The new scheme is simpler: one instance per thread, one instance per core. This ends up meaning fewer objects may be cached, but it also almost entirely eliminates contention between threads trying to rent/return objects. As a result, under heavy load it can actually do a better job of using pooled objects as it doesn't bail on using the cache in the face of contention. It also reduces concerns about larger machines being more negatively impacted by the caching. Under lighter load, since we don't cache as many objects, it does mean we may end up allocating a bit more, but generally not much more (and the size of the object we do allocate is a reference-field smaller).
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Diagnosers;
using System.Runtime.CompilerServices;
[MemoryDiagnoser]
public class Program
{
public static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
private const int Concurrency = 256;
private const int Iters = 100_000;
[Benchmark]
public Task NonPooling()
{
return Task.WhenAll(from i in Enumerable.Range(0, Concurrency)
select Task.Run(async delegate
{
for (int i = 0; i < Iters; i++)
await A().ConfigureAwait(false);
}));
static async ValueTask A() => await B().ConfigureAwait(false);
static async ValueTask B() => await C().ConfigureAwait(false);
static async ValueTask C() => await D().ConfigureAwait(false);
static async ValueTask D() => await Task.Yield();
}
[Benchmark]
public Task Pooling()
{
return Task.WhenAll(from i in Enumerable.Range(0, Concurrency)
select Task.Run(async delegate
{
for (int i = 0; i < Iters; i++)
await A().ConfigureAwait(false);
}));
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder))]
static async ValueTask A() => await B().ConfigureAwait(false);
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder))]
static async ValueTask B() => await C().ConfigureAwait(false);
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder))]
static async ValueTask C() => await D().ConfigureAwait(false);
[AsyncMethodBuilder(typeof(PoolingAsyncValueTaskMethodBuilder))]
static async ValueTask D() => await Task.Yield();
}
}
|
There was a problem hiding this comment.
LGTM!
It also reduces concerns about larger machines being more negatively impacted by the caching
To validate that you could use this template, modify it and run the benchmarks with and without your changes using the AMD (32 cores), ARM (48 cores), and Mono machine (56 cores).
...m.Private.CoreLib/src/System/Runtime/CompilerServices/PoolingAsyncValueTaskMethodBuilderT.cs
Outdated
Show resolved
Hide resolved
The current scheme caches one instance per thread in a ThreadStatic, and then has a locked stack that all threads contend on; then to avoid blocking a thread while accessing the cache, locking is done with TryEnter rather than Enter, simply skipping the cache if there is any contention. The locked stack is capped by default at ProcessorCount*4 objects. The new scheme is simpler: one instance per thread, one instance per core. This ends up meaning fewer objects may be cached, but it also almost entirely eliminates contention between threads trying to rent/return objects. As a result, under heavy load it can actually do a better job of using pooled objects as it doesn't bail on using the cache in the face of contention. It also reduces concerns about larger machines being more negatively impacted by the caching. Under lighter load, since we don't cache as many objects, it does mean we may end up allocating a bit more, but generally not much more (and the size of the object we do allocate is a reference-field smaller).
ghost
locked as resolved and limited conversation to collaborators
The current scheme caches one instance per thread in a ThreadStatic, and then has a locked stack that all threads contend on; then to avoid blocking a thread while accessing the cache, locking is done with TryEnter rather than Enter, simply skipping the cache if there is any contention. The locked stack is capped by default at ProcessorCount*4 objects.
The new scheme is simpler: one instance per thread, one instance per core. This ends up meaning fewer objects may be cached, but it also almost entirely eliminates contention between threads trying to rent/return objects. As a result, under heavy load it can actually do a better job of using pooled objects as it doesn't bail on using the cache in the face of contention. It also reduces concerns about larger machines being more negatively impacted by the caching. Under lighter load, since we don't cache as many objects, it does mean we may end up allocating a bit more, but generally not much more (and the size of the object we do allocate is a reference-field smaller).
This is on my 12-logical core box: