VisualStudioDiagnosticAnalyzerExecutorTests.TestCancellation is hanging

[jaredpar]

This is hanging on at least one Spanish run and is likely the cause of most of our Spanish hangs recently.

• https://dev.azure.com/dnceng/public/_build/results?buildId=815827&view=results
• https://dev.azure.com/dnceng/public/_build/results?buildId=816120&view=results
• https://dev.azure.com/dnceng/public/_build/results?buildId=814907&view=results

Using the dump file published with this test failure and following the async test hang guide I determined this was the cause.

Will send a PR to disable this test.

[CyrusNajmabadi]

FYI, this is likely a pri1 issue. Tagging @mavasani @sharwell

[CyrusNajmabadi]

(I'm not blocking disablign this. just that we need to investigate asap and address whatever is going on here).

[sharwell]

I wonder if this is fixed by ae59b76, or at minimum revealed by #47554. The current failure to validate asynchronous assertions is likely directly responsible for not immediately seeing the cause of this test failure.

[jaredpar]

Verified this is also the source of the Spanish hang here

https://dev.azure.com/dnceng/public/_build/results?buildId=816120

[jaredpar]

Verified this is also the source of a hang in Desktop 32 so it's not localized to Spanish.

https://dev.azure.com/dnceng/public/_build/results?buildId=814907&view=artifacts&type=publishedArtifacts

Basically this the cause of all our rolling build failures right now.

[mavasani]

This seems to be an actual product regression which is causing all out of proc diagnostics requests to hang, so active file diagnostics are not updated while you type. All diagnostic computation requests seemed to blocked PipeReaderStream.Read. Wondering if this is related to any recent broker service connection changes?

[mavasani]

Tagging @tmat

[mavasani]

Alright, so I have confirmed that this regression/race condition was introduced with Brokered services PR: #47216. Seems like there was a follow-up fix #47669, but I can still hit the hang on latest master bits.

I unskipped VisualStudioDiagnosticAnalyzerExecutorTests.TestCancellation and ran the test a couple of times on following different commits. NOTE: Test seems to always pass fine the first time it is executed, and hangs the second time.

1. Latest master branch - test hangs
2. Merge commit for brokered services PR: e53ffb7 - test hangs
3. Commit just prior to brokered services PR: 20760c9 - test runs fine

The stack at the time of the hang matches my screenshot in the above comment:

 	mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)	Unknown
 	mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)	Unknown
 	mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)	Unknown
 	[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]	
>	System.IO.Pipelines.dll!System.IO.Pipelines.PipeReaderStream.Read(byte[] buffer, int offset, int count) Line 51	C#
 	mscorlib.dll!System.IO.Stream.ReadByte()	Unknown
 	Nerdbank.Streams.dll!Nerdbank.Streams.FullDuplexStream.CombinedStream.ReadByte()	Unknown
 	Microsoft.CodeAnalysis.Workspaces.dll!Roslyn.Utilities.ObjectReader.TryGetReader(System.IO.Stream stream, bool leaveOpen, System.Threading.CancellationToken cancellationToken) Line 100	C#
 	Microsoft.CodeAnalysis.Features.dll!Microsoft.CodeAnalysis.Diagnostics.InProcOrRemoteHostAnalyzerRunner.ReadCompilerAnalysisResultAsync(System.IO.Stream stream, System.Collections.Generic.Dictionary<string, Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer> analyzerMap, Microsoft.CodeAnalysis.Diagnostics.DocumentAnalysisScope documentAnalysisScope, Microsoft.CodeAnalysis.Project project, System.Threading.CancellationToken cancellationToken) Line 215	C#
 	Microsoft.CodeAnalysis.Features.dll!Microsoft.CodeAnalysis.Diagnostics.InProcOrRemoteHostAnalyzerRunner.AnalyzeOutOfProcAsync.AnonymousMethod__2(System.IO.Stream stream, System.Threading.CancellationToken cancellationToken) Line 198	C#
 	Microsoft.CodeAnalysis.Remote.Workspaces.dll!Microsoft.CodeAnalysis.Remote.BrokeredServiceConnection<Microsoft.CodeAnalysis.Diagnostics.IRemoteDiagnosticAnalyzerService>.InvokeStreamingServiceAsync.AnonymousMethod__1() Line 144	C#
 	mscorlib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.Task<Microsoft.CodeAnalysis.Workspaces.Diagnostics.DiagnosticAnalysisResultMap<Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer, Microsoft.CodeAnalysis.Workspaces.Diagnostics.DiagnosticAnalysisResult>>>.InnerInvoke()	Unknown

Another easy way to repro it is just F5 latest Roslyn master bits, open any solution and type for a few seconds introducing compiler syntax errors. After few edits you will no longer see any diagnostics being updated in the IDE.