Timeout or Keep-Alive for open connection from HttpCompletionOption.ResponseHeadersRead ?

[Bio2hazard]

Hi,

we have noticed that some of the background workers on our services occasionally just stop.

Investigating the issue, I found that a library we use is using ResponseHeadersRead and that the workers stop because the target machine disappeared without explicitly closing the socket.

Our workers have stopped working for many hours ( before we inevitably restart them ), so it does not seem like these open connections have any timeouts or keep-alives on them.

My questions are:

1. Is there any way we can set a application-wide timeout or keep-alive or even maximum read time for these open stream connections?

2. Is there any way to abort an on-going read? I have tried disposing of stream, but the other thread remains blocked trying to read from the socket.

3. Can you offer some guidance or best practices around the problem? Streams are very abstract, and if all your method receives is a Stream, it's next to impossible to know how to safely work with the given stream. Furthermore, none of the convenience methods seem to support cancellation tokens, and any synchronous reads appear doomed to begin with.

   • HttpContent.ReadAsStringAsync ? No CTS Support => Application Deadlock Potential
   • HttpContent.ReadAsByteArrayAsync ? No CTS Support => Application Deadlock Potential
   • HttpContent.ReadAsStreamAsync > StreamReader.ReadToEndAsync ? No CTS Support => Application Deadlock Potential
   • HttpContent.ReadAsStreamAsync > StreamReader.ReadLineAsync ? No CTS Support => Application Deadlock Potential
   • HttpContent.ReadAsStreamAsync > Any synchronous call that reads from a stream => Application Deadlock Potential

It feels like the exact situation that Stream.CanTimeout , Stream.ReadTimeout and Stream.WriteTimeout was made for, but for some reason they are not supported either.

[davidsh]

It feels like the exact situation that Stream.CanTimeout , Stream.ReadTimeout and Stream.WriteTimeout was made for, but for some reason they are not supported either.

Those stream properties are legacy properties designed in the days before async/await/cancellationtokens. That mechanism for "timeout" is not supported since async streams use cancellationtoken as the mechanism mostly.

Furthermore, none of the convenience methods seem to support cancellation tokens, and any synchronous reads appear doomed to begin with.

Adding cancellation tokens to HttpClient and HttpContent methods is issue #916 and we plan to address it via API review.

[Bio2hazard]

Thank you for the answer.

Those stream properties are legacy properties designed in the days before async/await/cancellationtokens. That mechanism for "timeout" is not supported since async streams use cancellationtoken as the mechanism mostly.

I can't help but disagree a bit. The problem is that all we have is Stream. There is no explicit AsyncStream that ony exposes Async methods. Also, if you inherit from Stream, you are only required to provide implementations for synchronous read / write calls.

Don't get me wrong, I know it's to preserve compatibility, but so long as synchronous methods remain a first-class feature of Streams, a mechanism for timing out is needed.

Furthermore, consider this example:

using (var responseStream = await httpResponse.Content.GetStreamAsync())
using (var reader = new StreamReader(responseStream))
{
    responseStream.ReadTimeout = 5_000;
    return await reader.ReadToEndAsync();
}

A great feature of Streams is how they can be used to very cleanly build a near allocation-free pipeline of multiple transforms, readers and writers.

Often after setting up the pipeline you just want to run all available data through it, and that process is usually kicked off on the topmost level of the pipeline ( in the example that would be StreamReader ), which then is the only place and only opportunity where you could pass in the CancellationToken.

Setting the read timeout on the other hand allows for customizing the acceptable timeout anywhere in the pipeline. In the example given, I would know that responseStream involves networking and needs to have a timeout set, while StreamReader only acts on data in-memory and can't just stall.

The other problem is that the analyzers built into Visual Studio ( + StyleCop + FxCop ) do not recommend to use the Async variants over the synchronous variants, even when used inside an asynchronous method. They will recommend that you pass in a cancellation token, but they will not recommend that you switch from reader.ReadToEnd(); to await reader.ReadToEndAsync();.

It just seems exceedingly easy to accidentally lock up your entire application. It's also hell to debug since there are no exceptions, no errors, no microsoft messages of elevated severity to ilogger - nothing. The application doesn't fault either, the thread simply just stalls. None of the learning materials online take this situation into consideration either - in an attempt to be simple, they use the helper methods which as mentioned don't support cancellation. There are no warnings on the ResponseHeadersRead flag or on HttpContent.

[karelz]

Triage:

• HttpContent.ReadAsStringAsync and HttpContent.ReadAsByteArrayAsync are addressed in PR Implement HttpContent and HttpClient cancellation APIs #686 (in .NET 5).
• StreamReader overloads are tracked in Add CancellationToken to StreamReader.Read* methods #20824

Closing.