Crash when disposing an EventListener during shutdown

[kevingosse]

Description

When there are multiple EventListeners in the same application, disposing one of them during application shutdown causes a crash. It affects at least .NET 6/7/8. Reproduced on Windows and Linux.

Reproduction Steps

Minimal repro:

    public class MyEventListener : EventListener
    {
        public MyEventListener()
        {
            EventSourceCreated += (_, e) => EnableEventSource(e.EventSource);
        }

        private void EnableEventSource(EventSource eventSource)
        {
            EnableEvents(eventSource, EventLevel.Informational, EventKeywords.All);
        }
    }

    internal class Program
    {
        static EventListener listener1;
        static EventListener listener2;

        static async Task Main(string[] args)
        {
            listener1 = new MyEventListener();
            listener2 = new MyEventListener();


            AppDomain.CurrentDomain.ProcessExit += (_, _) => listener1.Dispose();
        }
    }

Expected behavior

The app exits normally.

Actual behavior

The app crashes when exiting:

Fatal error. Internal CLR error. (0x80131506)
   at System.Diagnostics.Tracing.EventPipeInternal.Enable(System.String, System.Diagnostics.Tracing.EventPipeSerializationFormat, UInt32, System.Diagnostics.Tracing.EventPipeProviderConfiguration[])
   at System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration()
   at System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener, System.Diagnostics.Tracing.EventCommand, Boolean, System.Diagnostics.Tracing.EventLevel, System.Diagnostics.Tracing.EventKeywords)
   at System.Diagnostics.Tracing.EventListener.DisableEvents(System.Diagnostics.Tracing.EventSource)
   at System.Diagnostics.Tracing.EventListener.CallDisableEventsIfNecessary(System.Diagnostics.Tracing.EventDispatcher, System.Diagnostics.Tracing.EventSource)
   at System.Diagnostics.Tracing.EventListener.RemoveReferencesToListenerInEventSources(System.Diagnostics.Tracing.EventListener)
   at System.Diagnostics.Tracing.EventListener.Dispose()
   at SimpleApp.Program+<>c.<Main>b__2_0(System.Object, System.EventArgs)

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[hoyosjs]

Crash stack:

00000001701426D0 0000000100EC8D30 libcoreclr.dylib!ep_session_start_streaming(_EventPipeSession*) + 36
0000000170142710 0000000100EC8CE0 libcoreclr.dylib!ep_start_streaming(unsigned long long) + 124
0000000170142740 0000000100C73CB4 libcoreclr.dylib!EventPipeInternal_Enable + 724
0000000170142830 0000000101FB0628
0000000170142850                  [InlinedCallFrame: 0000000170142850]
0000000170142850                  [InlinedCallFrame: 0000000170142850]
0000000170142830 0000000101FB0600 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeInternal.Enable(System.String, System.Diagnostics.Tracing.EventPipeSerializationFormat, UInt32, System.Diagnostics.Tracing.EventPipeProviderConfiguration[]) + 400
0000000170142960 0000000101FB7AB4 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() + 356
0000000170142A70 0000000101FB7890 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener, System.Diagnostics.Tracing.EventCommand, Boolean, System.Diagnostics.Tracing.EventLevel, System.Diagnostics.Tracing.EventKeywords) + 192
0000000170142AD0 0000000101FCC128 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallDisableEventsIfNecessary(System.Diagnostics.Tracing.EventDispatcher, System.Diagnostics.Tracing.EventSource) + 120
0000000170142B00 0000000101FCC210 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.RemoveReferencesToListenerInEventSources(System.Diagnostics.Tracing.EventListener) + 176
0000000170142B60 0000000101FCB724 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() + 260
0000000170142BA0 0000000102AC5934 shutdownEL.dll!Program+<>c.<Main>b__2_0(System.Object, System.EventArgs) + 68
FFFFFFFFFFFFFFFF 0000000101FB7AB4
FFFFFFFFFFFFFFFF 0000000101FB7890
FFFFFFFFFFFFFFFF 0000000101FCC128
FFFFFFFFFFFFFFFF 0000000101FCC210
FFFFFFFFFFFFFFFF 0000000101FCB724
FFFFFFFFFFFFFFFF 0000000102AC5934
FFFFFFFFFFFFFFFF 0000000100DC3444 libcoreclr.dylib!CallDescrWorkerInternal + 132
0000000170142BF0 0000000100C3E80C libcoreclr.dylib!MethodDescCallSite::CallTargetWorker(unsigned long long const*, unsigned long long*, int) + 856
0000000170142E60 0000000100B3908C libcoreclr.dylib!AppDomain::RaiseExitProcessEvent() + 168
0000000170142F60 0000000100C76A50 libcoreclr.dylib!FinalizerThread::FinalizerThreadStart(void*) + 116
0000000170142F90 0000000100B29A40 libcoreclr.dylib!CorUnix::CPalThread::ThreadEntry(void*) + 380
0000000170142FD0 00000001968DEF94 libsystem_pthread.dylib!_pthread_start + 136

ep_provider_callback_data_queue_try_dequeue fails inside ep_enable as ep_raise_error_if_nok (ep_volatile_load_eventpipe_state () == EP_STATE_INITIALIZED);. fails - the state is shutdown. We end up returning a null session and then AV when trying to stream. @davmason @noahfalk Should we guard EventPipeInternal_Enable to nullptr returned as the session? The problem is that's a QCall from EventPipeInternal.Enable and that comes from CommitDispatchConfiguration which will throw. There's nothing that handles this exception - not sure if there's a change that needs to be a little more thorough.

[kevingosse]

For what it's worth, I'm a bit stuck here: if I dispose the EventListener I risk running into this crash, so I changed the implementation in the Datadog tracer to stop disposing it. But now we randomly crash at shutdown with an access violation with this callstack:

System_Private_CoreLib!System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload+0x16f [/_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipePayloadDecoder.cs @ 51] 
System_Private_CoreLib!System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent+0x94 [/_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/NativeRuntimeEventSource.cs @ 55] 
System_Private_CoreLib!System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners+0xcc [/_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipeEventDispatcher.cs @ 175] 
System_Private_CoreLib!System.Threading.Tasks.Task.InnerInvoke+0x1e [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2397] 
System_Private_CoreLib!System.Threading.Tasks.Task.<>c.<.cctor>b__272_0+0x16 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2376] 
System_Private_CoreLib!System.Threading.ExecutionContext.RunInternal+0x69 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 183] 
System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0xb4 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2333] 
System_Private_CoreLib!System.Threading.Tasks.Task.ExecuteEntryUnsafe+0x53 [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 2277] 
System_Private_CoreLib!System.Threading.Tasks.ThreadPoolTaskScheduler.<>c.<.cctor>b__10_0+0x1a [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs @ 36] 
System_Private_CoreLib!System.Threading.Thread.StartCallback+0x61 [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs @ 105] 

Any idea of a workaround, no matter how dirty, that we could use to fix those crashes?

[davmason]

I can't think of a workaround here that can be done client side, but the fix is simple enough we should still be able to fix in 9. @mdh1418 would you be willing to look at this? The fix should be pretty simple, to add a check that sessionID is valid before using it here:

runtime/src/coreclr/vm/eventpipeinternal.cpp

Line 47 in 4fa45a9

EventPipeAdapter::StartStreaming(sessionID);

And then change it from throwing an exception here to just giving up and exiting:

runtime/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipeEventDispatcher.cs

Line 107 in 4fa45a9

throw new EventSourceException(SR.EventSource_CouldNotEnableEventPipe);