Using Process.BeginOutputReadLine can lead to lost stdout lines

[joelverhagen]

When writing code that creates child processes and needs to capture stdout, I have found that lines of output go missing the following condition:

• lots of parallelism...
• ProcessStartInfo.RedirectStandardOutput = true
• Process.OutputDataReceived += CollectLines (that is, using the async flow)
• Process.BeginOutputReadLine()
• Process.WaitForExit(int.MaxValue)

Note that passing -1 (infinite timespan) to WaitForExit does not cause lines go missing (based on my own testing).

I have created a little demo app to do this. It repro's on both netcoreapp1.0 and net45: processredirect.zip. Just use dotnet run to try it out.

That being said, you can capture stdout by using Process.StandardOutput (which is a StreamReader). However to consume this asynchronously and safely you need to use background threads/tasks. In other words:

stdout capture method	using infinite timeout	is flaky
StandardOutput StreamReader	yes	no
StandardOutput StreamReader	no	no
OutputDataReceived event	yes	no
OutputDataReceived event	no	yes

I think this a problem because it's pretty easy to step into this pitfall. Using the events to capture output is very convenient since you don't have to worry about making background threads or tasks. Also, the internet recommends this method:

• http://stackoverflow.com/questions/285760/how-to-spawn-a-process-and-capture-its-stdout-in-net
• https://msdn.microsoft.com/en-us/library/system.diagnostics.process.beginoutputreadline(v=vs.110).aspx

I have only tried this on Windows, .NET Framework 4.5 and .NET Core (netcoreapp1.0). Also, all of this information is related to stdout, but I imagine stderr has the same problem. My dotnet info is:

Microsoft .NET Core Shared Framework Host

  Version  : 1.0.1
  Build    : cee57bf6c981237d80aa1631cfe83cb9ba329f12

> dotnet --version
1.0.0-preview2-003131

[joelverhagen]

I've updated the test app to support non-Windows systems (by using echo):
processredirect2.zip.

The results are the same for Windows, Linux, and Mac OS X.

[stephentoub]

@joelverhagen, for better or worse, what you're seeing is documented design...

For Process.WaitForExit():
https://msdn.microsoft.com/en-us/library/fb4aw7b8(v=vs.110).aspx

This overload ensures that all processing has been completed, including the handling of asynchronous events for redirected standard output. You should use this overload after a call to the WaitForExit(Int32) overload when standard output has been redirected to asynchronous event handlers.

And for Process.WaitForExit(Int32):
https://msdn.microsoft.com/en-us/library/ty0d8k56(v=vs.110).aspx

When standard output has been redirected to asynchronous event handlers, it is possible that output processing will not have completed when this method returns. To ensure that asynchronous event handling has been completed, call the WaitForExit() overload that takes no parameter after receiving a true from this overload.

You can see here that the code explicitly does not wait for the redirected output to complete if it was passed a non-infinite timeout:

• https://referencesource.microsoft.com/#System/services/monitoring/system/diagnosticts/Process.cs,2458
• https://github.com/dotnet/corefx/blob/bffef76f6af208e2042a2f27bc081ee908bb390b/src/System.Diagnostics.Process/src/System/Diagnostics/Process.Windows.cs#L131

That said, I don't know why it was designed this way.

[Priya91]

@joelverhagen Using WaitForExit without timeout after the other overload should unblock you, as pointed out in the docs. Closing this issue, please reopen if further assistance needed.