Automatically capture heap dumps

[jamescrosswell]

Resolves #2580

Basic usage

samples/Sentry.Samples.Console.Basic/Sentry.Samples.Console.Basic.csproj includes an example of how to configure automatic heap dumps:

sentry-dotnet/samples/Sentry.Samples.Console.Basic/Program.cs

Lines 41 to 50 in ffad135

	// This option tells Sentry to capture a heap dump when the process uses more than 5% of the total memory. The heap
	// dump will be sent to Sentry as a file attachment.
	options.EnableHeapDumps(5);
	// This determines the level of heap dump events that are sent to Sentry
	options.HeapDumpEventLevel = SentryLevel.Warning;
	// A debouncer can be configured to tell Sentry how frequently to send heap dumps. In this case we've configured it
	// to capture a maximum of 3 events per day and to wait at least 1 hour between each event.
	options.HeapDumpDebouncer = Debouncer.PerDay(3, TimeSpan.FromHours(1));

That sample demonstrates using a built in trigger that we provide, that triggers heap dumps when memory usage exceeds a certain threshold (5% in that example).

Alternatively there is an override for the EnableHeapDumps method that SDK users can call to provide a custom trigger:

sentry-dotnet/src/Sentry/SentryOptions.cs

Line 561 in 265e397

public void EnableHeapDumps(HeapDumpTrigger trigger) => HeapDumpTrigger = trigger;

Prerequisites

dotnet-gcdump

The SDK relies on dotnet-gcdump to capture the heap dumps.

The dotnet-gcdump can be installed globally on the machine or container where the heap dumps will be captured by running:

dotnet tool install --global dotnet-gcdump

net6.0

dotnet-gcdump requires .NET 6 or later.

Analysing the Dump File

A couple of different options:

1. dotnet-gcdump report
2. Open the dump file with Perfview or Visual Studio... provides richer functionality but is only available on Windows.
3. dotnet-heapview

See Stefan Geiger's blog for a bit of info on the first two options.

Possible improvements

• According to the docs, the static GC.* methods we use to be notified of garbage collection events throw an exception when concurrent garbage collection is enabled so this may not currently work with Background GC mode
  • If/when server mode is enabled, we may have to revert to either creating a dangling object to detect GC events (not great as it only detects G0 collections) or alternatively some kind of timer
  • Particularly important as DATAS is the new default mode in net9.0 and we're not guaranteed to have a single heap in applications with DATAS configured
  • When experimenting with this, I didn't observer any InvalidOperationException when I attempted to enable Background GC... so we may need to see if any of our users are running into this problem (and can help us reproduce it) before we can fix it. Hopefully this is the kind of feedback we can gather during alpha testing.

[bruno-garcia]

CI has issues

[bruno-garcia]

@jamescrosswell so we prob shouldn't bundle stuff after all, lets document folks adding the CLI to the output?
Or can we download as part of the build or release process, directly from NuGet? (assuming that executable is inside a nuget package?)

[jamescrosswell]

@jamescrosswell so we prob shouldn't bundle stuff after all, lets document folks adding the CLI to the output?
Or can we download as part of the build or release process, directly from NuGet? (assuming that executable is inside a nuget package?)

OK, we could download it as part of the transient build file, but we'd still be shipping it along with customer's applications in that case... which I don't think resolves any legal concerns.

I think we just advise customer that they need to install the tool globally on any machines where they want to capture heap dumps... and of course we can log a warning/error if dotnet-gcdump isn't installed.

[bruno-garcia]

I think we just advise customer that they need to install the tool globally on any machines where they want to capture heap dumps... and of course we can log a warning/error if dotnet-gcdump isn't installed.

OK that's reasonable. And also a good first step. We can always improve from there

[bruno-garcia]

Releasing this branch as 5.0.0-alpha.2, merging back on itself:

getsentry/publish#4608

[bruno-garcia]

Need to wait for this to be merged, and a new craft release:

• Install .NET 9 SDK craft#564

[jamescrosswell]

If/when server mode is enabled, we may have to revert to either creating a dangling object to detect GC events (not great as it only detects G0 collections) or alternatively some kind of timer

If we do need to do this, the DanglingGarbageCollectionMonitor is fairly simple:

namespace Sentry.Internal;

/// <summary>
/// Simple class to determine when GarbageCollection occurs
/// </summary>
internal sealed class DanglingGarbageCollectionMonitor
{
    private readonly CancellationToken _cancellationToken;
    private readonly Action _onGarbageCollected;

    private DanglingGarbageCollectionMonitor(Action onGarbageCollected, CancellationToken cancellationToken)
    {
        _cancellationToken = cancellationToken;
        _onGarbageCollected = onGarbageCollected;
    }

    ~DanglingGarbageCollectionMonitor()
    {
        // If monitoring has stopped or the App is shutting down, stop monitoring.
        var stopped = _cancellationToken.IsCancellationRequested;
        if (stopped || Environment.HasShutdownStarted || AppDomain.CurrentDomain.IsFinalizingForUnload())
        {
            return;
        }

        // Every time this class gets cleaned up by the GC, we create another dangling instance of the class (which will
        // be cleaned up in the next GC cycle)... so we keep creating dangling references as long as we want to keep
        // monitoring.
        CreateDanglingMonitor(_onGarbageCollected, _cancellationToken);
        _onGarbageCollected.Invoke();
    }

    private static void CreateDanglingMonitor(Action onGarbageCollected, CancellationToken cancellationToken)
    {
        // ReSharper disable once ObjectCreationAsStatement
#pragma warning disable CA1806
        new DanglingGarbageCollectionMonitor(onGarbageCollected, cancellationToken);
#pragma warning restore CA1806
    }

    public static void Start(Action onGarbageCollected, CancellationToken cancellationToken = default) =>
        CreateDanglingMonitor(onGarbageCollected, cancellationToken);
}

And we could alter the GarbageCollectionMonitor to leverage this like so:

namespace Sentry.Internal;

/// <summary>
/// Simple class to detect when Full Garbage Collection occurs
/// </summary>
internal sealed class GarbageCollectionMonitor
{
    private const int MaxGenerationThreshold = 10;
    private const int LargeObjectHeapThreshold = 10;

    public static Task Start(Action onGarbageCollected, CancellationToken cancellationToken, IGCImplementation? gc = null) =>
        Task.Run(() => MonitorGarbageCollection(onGarbageCollected, cancellationToken, gc), cancellationToken);

    private static void MonitorGarbageCollection(Action onGarbageCollected, CancellationToken cancellationToken, IGCImplementation? gc = null)
    {
        try
        {
            MonitorGarbageCollectionNotifications(onGarbageCollected, cancellationToken, gc);
        }
        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
        {
            // Ignore
        }
        catch (InvalidOperationException)
        {
            // If concurrent GC is enabled, methods like GC.RegisterForFullGCNotification will throw an
            // <see cref="InvalidOperationException"/> so we need to use another mechanism to monitor the GC
            DanglingGarbageCollectionMonitor.Start(onGarbageCollected, cancellationToken);
        }
    }

    private static void MonitorGarbageCollectionNotifications(Action onGarbageCollected, CancellationToken cancellationToken, IGCImplementation? gc = null)
    {
        gc ??= new SystemGCImplementation();
        try
        {
            gc.RegisterForFullGCNotification(MaxGenerationThreshold, LargeObjectHeapThreshold);
            while (!cancellationToken.IsCancellationRequested)
            {
                if (gc.WaitForFullGCComplete(TimeSpan.FromSeconds(1)) == GCNotificationStatus.Succeeded)
                {
                    onGarbageCollected?.Invoke();
                }
            }
        }
        catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
        {
            // Ignore
        }
        finally
        {
            gc.CancelFullGCNotification();
        }
    }
}

[jamescrosswell]

There may be an even better way to detect GC events, that only triggers for G2 events, using GC.ReRegisterForFinalize.

There's an example of how to use this in the dotnet runtime source code:
https://github.com/dotnet/runtime/blob/2ac0743b18cf70cc6047f0ec35d3a7a6a4428354/src/libraries/System.Private.CoreLib/src/System/Gen2GcCallback.cs