SslStream object disposed exception #28171

halter73 · 2018-12-12T20:19:56Z

From @timmydo on Saturday, 08 December 2018 21:55:49

I'm not sure if this is the right place to report this, but I'm seeing this exception occasionally:

kubernetes docker pod uname -a:
Linux platform-prod-577cf48b4b-wrg7x 4.15.0-1030-azure dotnet/corefx#31~16.04.1-Ubuntu SMP Tue Oct 30

context:
running kestrel behind nginx on linux in docker image based on dotnet:2.1-aspnetcore-runtime

19:40:01 UTC 2018 x86_64 GNU/Linux
Unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

pod error code:

Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Fri, 07 Dec 2018 17:06:46 -0800
      Finished:     Fri, 07 Dec 2018 23:48:04 -0800

let me know if there is more info I could provide. thanks.

Copied from original issue: aspnet/KestrelHttpServer#3111

The text was updated successfully, but these errors were encountered:

halter73 · 2018-12-12T20:19:57Z

From @halter73 on Wednesday, 12 December 2018 20:16:44

System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)

The ODE seems to have happened during TLS renegotiation. Kestrel ensures that it's done with its SslStreams before disposing them meaning that any calls to AuthenticateAsServerAsync(), ReadAsync(), WriteAsync(), etc... have completed.

@timmydo Do you have any evidence that the SslStream belongs to Kestrel? Kestrel doesn't initiate a renegotiations itself, and I don't know why a client would do so either. I suspect that HttpClient might own the SslStream.

Either way, we should probably move this issue to dotnet/corefx.

halter73 · 2018-12-12T20:20:40Z

@karelz @Tratcher

karelz · 2018-12-12T20:46:57Z

How often does it happen?
Does it happen on more than 1 machine or for more than 1 user?
It does not seem that the above info is sufficient to root cause it, or to even identify the components owning the SslStream.

timmydo · 2018-12-12T23:49:56Z

this seems to happen on different machines. looks like it's happening from an http client request because the program isn't listening for connections (it's just processing things in a queue--so not kestrel related like the original issue). maybe a handful of times a day. if there is a way you could recommend I get more info that would be great. if it's not easy, I can just close this. I agree the stack trace doesn't look that helpful

karelz · 2019-01-09T02:15:47Z

I don't know about a way forward without additional info or simple repro. Closing.
We can reopen if more people are affected or when there is actionable info available

timmydo · 2019-01-09T02:37:24Z

@karelz This issue occurs regularly (every minute) if there is a lot of traffic. I tried ramping up some to this new environment today and had to take it down due to crashes. The stack trace usually appears at the end of the logs, so I would suspect that it's fatal whatever it is. I will try to narrow down which dependency is at fault, but like I said above, I suspect it's related to the http client.

Do you have any ideas on how I could get more info?

timmydo · 2019-01-09T18:05:00Z

Could this be related to #21330 ?

timmydo · 2019-01-10T01:44:42Z

Or #28073 and maybe fixed with dotnet/corefx#34089

LucasFarley · 2019-01-20T17:55:04Z

I facing the same problem and due that the application is closing.

Unhandled Exception: 
Unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Aborted (core dumped)

timmydo · 2019-01-21T01:22:09Z

3.0 daily builds fixed it for me

LucasFarley · 2019-01-21T12:23:04Z

@timmydo, where can i download this update?

davidsh · 2019-01-21T16:44:43Z

@timmydo, where can i download this update?

Daily builds of .NET Core are here: https://github.com/dotnet/core/blob/master/daily-builds.md

Port fix from PR dotnet#34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

davidsh · 2019-07-13T23:45:54Z

Re-opening for release/2.1 LTS port

Port fix from PR dotnet#34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

…ler (#39456) Port fix from PR #34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

davidsh · 2019-07-16T20:33:07Z

Fix for release/2.1 checked in with PR dotnet/corefx#39456

benjzh · 2019-08-07T14:13:21Z

When the fix will be released in .Net Core 2.2.(7?) officially? I see that it is already merged with the release branch several weeks ago.
Also it would be nice if somebody had any idea of a possible temporary workaround. (I tried and failed..)

davidsh · 2019-08-08T17:13:37Z

The fix will ship with the next 2.2.x release which is 2.2.7. And that will be sometime next month.

It will also ship with the next 2.1.x release which is 2.1.13. That should be shipped the same time as 2.2.x.

kyschouv · 2019-09-23T20:58:25Z

This doesn't seem to be fixed in .NET Core 3.0.0-preview9. Should I expect it to have been fixed in that release, or would it only appear in rc1 or the full release? (though mine is from an HttpClient)

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: The read operation failed, see inner exception.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslStream.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslStream.HandleQueuedCallback(Object& queuedStateRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

davidsh · 2019-09-23T21:01:05Z

.NET Core 3.0 released today. I suggest you try out the official release. If you still have a repro using .NET Core 3.0 and see these ObjectDisposed exceptions, please open a new issue and reference this one. Thanks!

kyschouv · 2019-09-23T21:20:02Z

Will do. Time to failure was about 9 days, so it may not happen again soon, but wanted to make sure to report it, since I jumped to 3.0 just for this fix.

kyschouv · 2019-09-27T19:34:34Z

We're getting the same issue on 3.0.0.

karelz · 2019-09-30T16:36:30Z

@kyschouv looks like the problem has multiple root causes (or similar symptoms)) and you've hit another one. Please file a new issue with description, call stacks, repro attempts info, etc., so that we can chase it down.

menozz · 2019-10-10T10:55:00Z

We have the same issue in high-load stress environment (release v3.0.0):

System.IO.IOException: The read operation failed, see inner exception.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)

ThatBlokeCalledJay · 2019-10-14T14:59:58Z

We have also been receiving this in our production environment, intermittently over the past few weeks. After reading through this issue, plus a couple of linked issues am able to repro consistently. In short, if I start uploading a large-ish file, one that allows me enough time to close the browser before the file upload completes, I get this error.

System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.CheckThrow(bool authSuccessCheck, bool shutdownCheck)
at System.Net.Security.SslState.get_SecureStream()
at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory source)
at System.Net.Http.HttpConnection.FlushAsync()

srdmnc · 2019-10-25T02:56:52Z

This issue is closed already. Where's the final fix?

davidsh · 2019-11-01T19:56:48Z

n short, if I start uploading a large-ish file, one that allows me enough time to close the browser before the file upload completes, I get this error.

What browser are you using? How is .NET Core involved with your browser exactly?

karelz · 2019-11-01T21:57:26Z

@srdmnc see https://github.com/dotnet/corefx/issues/34033#issuecomment-519608439

davidsh · 2019-11-01T22:08:25Z

We definitely fixed the original issue where an ObjectDisposedException on SslStream was being thrown on a threadpool thread and thus crashing the process with an un-catchable exception. That was fixed in .NET Core 3.0 as I mentioned.

However, it appears based on these reports above that there is still an ObjectDisposedException surfacing from SslStream during an HTTP request cancellation/timeout. This is showing up in the callstacks above as an inner exception to the TaskCanceledException. So, if you have a repro for that, please open a new issue so that can be investigated.

Keep in mind that the top-level exception, TaskCanceledException is probably correct. Something caused the HttpClient to cancel the request, most likely due to the HttpClient.Timeout occurring. However, there is potentially a bug still in .NET Core where the cancellation logic and the closing of the SslStream and TCP connections is throwing these inner exceptions that are surfacing in the call stack. Those exceptions can probably be trapped in a better way since the ObjectDisposedException is most likely "expected" as the connection is being closed due to the cancellation.

So, to help isolate this particular issue, please open a new issue and post repro instructions so we can diagnose. Thanks!

halter73 self-assigned this Dec 12, 2018

karelz unassigned halter73 Dec 12, 2018

karelz closed this as completed Jan 9, 2019

davidsh referenced this issue in davidsh/corefx Jul 13, 2019

Catch exception in ssl handshake so it can be propagated to async caller

273cfe9

Port fix from PR dotnet#34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

davidsh reopened this Jul 13, 2019

davidsh self-assigned this Jul 13, 2019

davidsh referenced this issue in davidsh/corefx Jul 13, 2019

Catch exception in ssl handshake so it can be propagated to async caller

823d84c

Port fix from PR dotnet#34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

davidsh referenced this issue in dotnet/corefx Jul 16, 2019

Catch exception in ssl handshake so it can be propagated to async cal…

dd39544

…ler (#39456) Port fix from PR #34089 (.NET Core 3.0) to release/2.1 LTS branch. Fixes #34033

davidsh closed this as completed Jul 16, 2019

msftgits transferred this issue from dotnet/corefx Feb 1, 2020

msftgits added this to the 2.1.x milestone Feb 1, 2020

ghost locked as resolved and limited conversation to collaborators Dec 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SslStream object disposed exception #28171

SslStream object disposed exception #28171

halter73 commented Dec 12, 2018

halter73 commented Dec 12, 2018

halter73 commented Dec 12, 2018

karelz commented Dec 12, 2018

timmydo commented Dec 12, 2018

karelz commented Jan 9, 2019

timmydo commented Jan 9, 2019

timmydo commented Jan 9, 2019

timmydo commented Jan 10, 2019

LucasFarley commented Jan 20, 2019 •

edited by karelz

Loading

timmydo commented Jan 21, 2019

LucasFarley commented Jan 21, 2019

davidsh commented Jan 21, 2019

davidsh commented Jul 13, 2019

davidsh commented Jul 16, 2019

benjzh commented Aug 7, 2019

davidsh commented Aug 8, 2019 •

edited

Loading

kyschouv commented Sep 23, 2019 •

edited

Loading

davidsh commented Sep 23, 2019 •

edited

Loading

kyschouv commented Sep 23, 2019

kyschouv commented Sep 27, 2019

karelz commented Sep 30, 2019

menozz commented Oct 10, 2019 •

edited

Loading

ThatBlokeCalledJay commented Oct 14, 2019

srdmnc commented Oct 25, 2019

davidsh commented Nov 1, 2019

karelz commented Nov 1, 2019

davidsh commented Nov 1, 2019

SslStream object disposed exception #28171

SslStream object disposed exception #28171

Comments

halter73 commented Dec 12, 2018

halter73 commented Dec 12, 2018

halter73 commented Dec 12, 2018

karelz commented Dec 12, 2018

timmydo commented Dec 12, 2018

karelz commented Jan 9, 2019

timmydo commented Jan 9, 2019

timmydo commented Jan 9, 2019

timmydo commented Jan 10, 2019

LucasFarley commented Jan 20, 2019 • edited by karelz Loading

timmydo commented Jan 21, 2019

LucasFarley commented Jan 21, 2019

davidsh commented Jan 21, 2019

davidsh commented Jul 13, 2019

davidsh commented Jul 16, 2019

benjzh commented Aug 7, 2019

davidsh commented Aug 8, 2019 • edited Loading

kyschouv commented Sep 23, 2019 • edited Loading

davidsh commented Sep 23, 2019 • edited Loading

kyschouv commented Sep 23, 2019

kyschouv commented Sep 27, 2019

karelz commented Sep 30, 2019

menozz commented Oct 10, 2019 • edited Loading

ThatBlokeCalledJay commented Oct 14, 2019

srdmnc commented Oct 25, 2019

davidsh commented Nov 1, 2019

karelz commented Nov 1, 2019

davidsh commented Nov 1, 2019

LucasFarley commented Jan 20, 2019 •

edited by karelz

Loading

davidsh commented Aug 8, 2019 •

edited

Loading

kyschouv commented Sep 23, 2019 •

edited

Loading

davidsh commented Sep 23, 2019 •

edited

Loading

menozz commented Oct 10, 2019 •

edited

Loading