Memory leak when serializing large string properties with System.Text.Json

[MathiasZander]

Description:

I've encountered a significant memory issue when serializing large string properties using the built-in System.Text.Json serializer in ASP.NET Core. This suggests a potential memory leak.

In the larger context of my application, out of over 50,000 messages, only 3 messages contained large strings (exceeding 20MB in size). This resulted in the memory consumption jumping from around 300 MB to several gigabytes. When I switched to using JSON.NET as the serializer for ASP.NET, this problem did not manifest. Also, when I limited the message size to a maximum of 100,000 characters, the issue disappeared.

Here's a simplified code snippet that reproduces the issue:

[HttpGet]
public IEnumerable<LogMessage> Get()
{
    var message = new StringBuilder();
    for (int i = 0; i < 10_000; i++)
    {
        message.AppendLine(Guid.NewGuid().ToString());
    }

    var logMessage = new LogMessage
    {
        Message = message.ToString()
    };
    var logMessages = new List<LogMessage>();

    for (int i = 0; i < 100; i++)
    {
        logMessages.Add(logMessage);
    }

    return logMessages;
}

public class LogMessage
{
    public string Message { get; set; }
}

Observations:

• After each call to the Get method, there's a noticeable increase in memory consumption.
• When I swap the values 10_000 and 100, the memory consumption remains relatively constant.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description:

I've encountered a significant memory issue when serializing large string properties using the built-in System.Text.Json serializer in ASP.NET Core. This suggests a potential memory leak.

In the larger context of my application, out of over 50,000 messages, only 3 messages contained large strings (exceeding 20MB in size). This resulted in the memory consumption jumping from around 300 MB to several gigabytes. When I switched to using JSON.NET as the serializer for ASP.NET, this problem did not manifest. Also, when I limited the message size to a maximum of 100,000 characters, the issue disappeared.

Here's a simplified code snippet that reproduces the issue:

[HttpGet]
public IEnumerable<LogMessage> Get()
{
    var message = new StringBuilder();
    for (int i = 0; i < 10_000; i++)
    {
        message.AppendLine(Guid.NewGuid().ToString());
    }

    var logMessage = new LogMessage
    {
        Message = message.ToString()
    };
    var logMessages = new List<LogMessage>();

    for (int i = 0; i < 100; i++)
    {
        logMessages.Add(logMessage);
    }

    return logMessages;
}

public class LogMessage
{
    public string Message { get; set; }
}

Observations:

• After each call to the Get method, there's a noticeable increase in memory consumption.
• When I swap the values 10_000 and 100, the memory consumption remains relatively constant.

Author:	MathiasZander
Assignees:	-
Labels:	area-System.Text.Json
Milestone:	-

[eiriktsarpalis]

Does the issue persist if you pre-cache your logMessages value?

[MathiasZander]

The standard time frame for a log query is cached, so no database query is being executed in this scenario.

[eiriktsarpalis]

Could you share a minimal repro? Ideally it should be a console app, but an aspnetcore app without DB access should work.

[MathiasZander]

I created this Repo

Also, here is an older image from a memory spike:

[eiriktsarpalis]

I notice that the image concerns IAsyncEnumerable serialization, which is very different in behavior compared to regular IEnumerable serialization that the repro is doing.

[MathiasZander]

I tried many things before I found the suspected fault, this was one of them.

[eiriktsarpalis]

I've been able to narrow down the reproduction to this:

using System.Runtime;
using System.Text.Json;

Console.WriteLine($"IsServerGC: {GCSettings.IsServerGC}");

var largeString = new string('x', 400_000);
var enumerable = Enumerable.Repeat(largeString, 300);

while (true)
{
    var stream = new MemoryStream();
    await JsonSerializer.SerializeAsync(stream, enumerable);
    // GC.Collect();
}

Like your example, this will show an almost unbounded increase in process memory. It seems to only happen with server GC turned on (which it is in the case of aspnetcore apps), once I revert to workstation mode or if I force a collection on each iteration the problem goes away.

Once I modified the code to reuse the same underlying MemoryStream:

using System.Runtime;
using System.Text.Json;

Console.WriteLine($"IsServerGC: {GCSettings.IsServerGC}");

var largeString = new string('x', 400_000);
var enumerable = Enumerable.Repeat(largeString, 300);
var stream = new MemoryStream();

while (true)
{
    await JsonSerializer.SerializeAsync(stream, enumerable);
    stream.Position = 0;
}

the issue immediately went away, indicating that this is not caused by some leak in System.Text.Json; it instead appears to relate to how the GC handles collection of LOH buffers created by the MemoryStream allocations.

If I were to simplify this further:

using System.Runtime;

Console.WriteLine($"IsServerGC: {GCSettings.IsServerGC}");

while (true)
{
    var bytes = new byte[120_000_000];
    bytes.AsSpan().Fill(0x42);
} 

I get identical behavior.

I'd be inclined to write this under by-design behavior of Server GC mode, but I defer to @Maoni0 on that. It isn't clear to me why this could be caused in the context of aspnetcore endpoints, but if I were to make an educated guess it might because the required buffers exceed the maximum size of the buffer pool being used, resulting in them being allocated on each request. @davidfowl might know.

[davidfowl]

This is by mostly design as your char[] is landing on the LOH which means it'll be collected when Gen2s happen.

Read the ASP.NET Core performance recommendations doc:

1. https://learn.microsoft.com/en-us/aspnet/core/fundamentals/best-practices?view=aspnetcore-7.0#minimize-large-object-allocations
2. https://learn.microsoft.com/en-us/aspnet/core/fundamentals/best-practices?view=aspnetcore-7.0#avoid-reading-large-request-bodies-or-response-bodies-into-memory

Outside of this, these are the related problems on the ASP.NET Core side:

• Allow server Memory Pool to shrink #27394
• Investigate performance of large writes on Kestrel #31110

cc @BrennanConroy @adityamandaleeka

[eiriktsarpalis]

Thanks, looks like we can close this over the linked issues.

[eiriktsarpalis]

as your char[] is landing on the LOH which means it'll be collected when Gen2s happen.

FWIW there's a leak happening in byte[] buffers as well, it's also observable if you cache the returned value:

using System.Text;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

var messages = GetMessages();
app.MapGet("/messages", () => messages);
app.Run();

static List<LogMessage> GetMessages()
{
    var message = new StringBuilder();
    for (int i = 0; i < 10_000; i++)
    {
        message.AppendLine(Guid.NewGuid().ToString());
    }

    var logMessage = new LogMessage
    {
        Message = message.ToString()
    };
    var logMessages = new List<LogMessage>();


    for (int i = 0; i < 100; i++)
    {
        logMessages.Add(logMessage);
    }

    return logMessages;
}

public class LogMessage
{
    public string Message { get; set; }
}

[davidfowl]

@eiriktsarpalis can you show the trace/dump/profile? The pathological case is when a single JSON token is big because it needs to be fully buffered by the JSON serializer. The JSON serializer has its own buffer and Kestrel has it's own buffer (this is one of the reasons I still want this API) and data is copied into 4K kestrel buffers that stick around forever but are reused for other requests.

[Alexander-Muylaert]

I am very interested in seeing this solved or contributing. I have a massive problem with this in combination with Azure functions with limited memory. I'm practically always OOM and it is completely unworkable. I have spend the past three weeks debugging, tracing and trying workarounds. I'm a bit at a death end now.

kind regards

a

[eiriktsarpalis]

The pathological case is when a single JSON token is big because it needs to be fully buffered by the JSON serializer.

I don't think that's the root cause in this case, from the previous examples compare the console app:

Console.WriteLine($"IsServerGC: {GCSettings.IsServerGC}");
var enumerable = Enumerable.Repeat(new string('x', 400_000), 300);

var stream = new MemoryStream();
while (true)
{
    await JsonSerializer.SerializeAsync(stream, enumerable);
    stream.Position = 0;
}

with the equivalent aspnetcore application:

Console.WriteLine($"IsServerGC: {GCSettings.IsServerGC}");
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

var enumerable = Enumerable.Repeat(new string('x', 400_000), 300);
app.MapGet("/messages", () => enumerable);
app.Run();

Using server GC, the first case shows constant (albeit increased) memory usage whereas the latter will demonstrate unbounded increase the more that endpoing is being hit.

[davidfowl]

Thanks @eiriktsarpalis

[BrennanConroy]

var enumerable = Enumerable.Repeat(new string('x', 400_000), 300);

var stream = new MemoryStream();
while (true)
{
    await JsonSerializer.SerializeAsync(stream, enumerable);
    stream.Position = 0;
}

This example isn't a fair apples-to-apples comparison. Json uses ArrayPool<T>.Shared which uses a [ThreadLocal] for the bucket of pooled arrays. And since this example is using MemoryStream the code never actually goes async and thus stays on the same thread the entire time. If you add await Task.Yield() in the loop you will start to see a higher working set due to a new thread occasionally picking up the work and so an empty bucket of pooled arrays will be hit.

the latter will demonstrate unbounded increase the more that endpoing is being hit.

There is not an unbounded increase in memory. The memory on my 16 core machine peaks around 700MB with Server GC.
And if I use Workstation GC it peaks around 460MB due to the GC actually doing work now. (Remember Server GC sees lots of available memory on your machine and happily uses it)

If you look at what memory is being used, you can see that Json uses the classic doubling array sizes when it needs more memory and it seems to get to and stay at a steady state of 16MB byte[] while writing the large enumerable. It starts at 2MB due to the 400,000 size string * 3 for max transcoding size which is 1.2m and array buckets are in power of 2 sizes, so 2MB. So Json allocates 2 + 4 + 8 + 16 = 30MB from the array pool every time it writes the full enumerable. 30MB * 16 cores = 480MB which is roughly the max memory our process will consume just from the JSON code.

Another semi-big chunk of memory (~40MB) when using Server GC is from System.IO.Pipelines.BufferSegment which is from Kestrel due to the internal pooling done by Pipes which is sadly thrown away when the connection is closed. See dotnet/runtime#49259 for some details. But this is all dead memory that the GC can cleanup when it feels like it. The rest of the memory is small in comparison and not worth looking for this discussion.

I did a quick spike of dotnet/runtime#68586 which David mentioned earlier in this thread and it shows significant improvements in working set when writing large payloads. In short with Server GC and Json writing directly to the Pipe results in a peak of roughly 140MB. This is largely due to Json only getting a 2MB buffer from the array pool when writing instead of doubling all the way up to 16MB. I didn't look too deeply into the Json code to figure out why it grabs such big arrays from the array pool, but this might be a potential improvement to be looked into?

I think we can close this issue. And potentially open a new issue against Json to not allocate such large arrays? And I'll follow up with some more detailed analysis of Json + Pipes in dotnet/runtime#68586.

[eiriktsarpalis]

the latter will demonstrate unbounded increase the more that endpoing is being hit.

There is not an unbounded increase in memory.

For whatever reason, I also can't reproduce an unbounded increase right now. I must have made a mistake back then.

I think we can close this issue. And potentially open a new issue against Json to not allocate such large arrays?

Ultimately I think it all boils down to addressing this issue: dotnet/runtime#67337

[BrennanConroy]

Closing per discussion above. There isn't a leak, and there are other issues filed to improve these scenarios.

[depler]

Seems to be related: #55490