SignalR Hub failing to send a message due to connection problems #675
Comments
|
Thanks for reporting this @dmitrysamuylovpharo. You're correct - the SDK on app server side will automatically reconnect dropped connections. |
|
@KKhurin Sure, I'll try updating to the latest SDKs and monitor for a bit, I'll add another post here once that's done with any news either way. |
|
@KKhurin Hi, sorry for the delay, I updated all SDKs, in the web application hosting the SignalR hub and in the mobile clients to the latests versions. I still see the problem happening occasionally with a very similar pattern with several instances of this error each time it happens (I presume one for each client connected at the time):
Callstack: What else can be done to debug this further to figure out why this is happening and not automatically reconnecting? Any suggestions? |
|
Thank you @dmitrysamuylovpharo for trying the latest SDK. It should automatically reconnect after getting this error so I'll have to dig deeper to see if there is a potential for not reconnecting. I'll get back with the results of the investigation. One thing to note is that in the current design messages can queue up for a particular service connection (if the App Server is sending them faster than the service can receive them) and once the connection is dropped each of these queued messages will fail with this error. In case when the number of the queued up messages is very large it may take a long time to log errors for each of them and thus create an impression that we keep using the same connection (while in fact the newly sent messages are already using the new healthy connection). We are aware of this problem and have plans to address this in the future releases (along with salvaging those queued message). In my tests this however only happens when the app servers manage to severely overwhelm the service. Could this be your case? |
|
@KKhurin no, our message traffic is very low, can't possibly be this scenario, we only send a message to all the clients in response to an occasional event manually triggered by a user's action, this only happens a handful of times a day. |
|
@dmitrysamuylovpharo thank you for the confirmation. I'll look further into it. |
|
@KKhurin today got the same type of error scenario but now it is returning a slightly different error message, not sure if that's of any help:
|
|
@dmitrysamuylovpharo this indicates an error on the service side. We can take a look at the service logs to see what is causing it - if interested in getting to the bottom of this please email the time when it happened and your "ResourceId" to <my_git_alias>@microsoft.com. However more importantly I am finally seeing a bug on SDK side which would lead to the App Server to keep using the faulted connection for some period of time after it gets an error from the service. The fix should be on its way shortly. |
…e of the service connection (Azure#763)
|
I believe this specific issue is already addressed. Please feel free to reopen the issue if you still have concerns about it. |
We are running into a problem of failed SignalR message sends from within an Azure hosted WebSite based asp.net core web-service. This web-service also implements and hosts the SignalR.Hub which connects to an Azure SignalR instance. From within one of the controllers in this web-service after performing some operations we try to send a SignalR message to notify all connected clients that an important operation has been completed and they can handle this accordingly. This message send periodically fails in a way that is reminiscent of several issues reported on here in the past, I hope someone can help me identify where the problem lies and how to prevent this from happening.
I understand that disconnects can and do occur from time to time, but the SignalR connection will automatically reconnect, if I understand correctly. I saw mention/recommendation for the clients to have handling for re-connecting and we have that in place on all our clients, this seems to be functioning ok. In this case the connection issue seems to be within the SignalR Hub to the Azure SignalR
Here are the errors I'm seeing in my Application Insights for our web-service that hosts the SignalR Hub at the time of one of these failed message sends:
1.MoveNext","level":3,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":4,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":5,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":6,"line":0},{"assembly":"System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Security.SslStreamInternal+<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d1.MoveNext","level":7,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":8,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":9,"line":0},{"assembly":"Microsoft.AspNetCore.Http.Connections.Client, Version=1.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport+d__20.MoveNext","level":10,"line":0}],"outerId":"0","message":"Error while sending a message.","severityLevel":"Error","type":"System.IO.IOException","id":"14093239"},{"outerId":"14093239","message":"An existing connection was forcibly closed by the remote host","severityLevel":"Error","type":"System.Net.Sockets.SocketException","id":"33060543"}]Next it reported a few instances of this error:
2.MoveNext","level":0,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":1,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":2,"line":0},{"assembly":"Microsoft.AspNetCore.Http.Connections.Client, Version=1.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport+<StartReceiving>d__19.MoveNext","level":3,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":4,"line":0},{"assembly":"System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51","method":"System.IO.Pipelines.PipeCompletion.ThrowLatchedException","level":5,"line":0},{"assembly":"System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51","method":"System.IO.Pipelines.Pipe.GetReadResult","level":6,"line":0},{"assembly":"System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51","method":"System.IO.Pipelines.Pipe.GetReadAsyncResult","level":7,"line":0},{"assembly":"System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51","method":"System.IO.Pipelines.Pipe+DefaultPipeReader.GetResult","level":8,"line":0},{"assembly":"Microsoft.Azure.SignalR.Common, Version=1.0.7.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.Azure.SignalR.ServiceConnectionBase+<ProcessIncomingAsync>d__53.MoveNext","level":9,"line":0}],"outerId":"0","message":"Connection 8f916f18-2d11-496a-bdff-a9dad81eb434 to the service was dropped.","severityLevel":"Error","type":"System.Net.WebSockets.WebSocketException","id":"3247361"},{"parsedStack":[{"assembly":"System.Net.Sockets, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException","level":0,"line":0},{"assembly":"System.Net.Sockets, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.GetResult","level":1,"line":0},{"assembly":"System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Security.SslStreamInternal+<<FillBufferAsync>g__InternalFillBufferAsync|38_0>d1.MoveNext","level":2,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":3,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":4,"line":0},{"assembly":"System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Security.SslStreamInternal+d__341.MoveNext","level":5,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":6,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":7,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":8,"line":0},{"assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Http.HttpConnection+<ReadBufferedAsyncCore>d__95.MoveNext","level":9,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":10,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":11,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":12,"line":0},{"assembly":"System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.Http.HttpConnection+RawConnectionStream+<ReadAsync>d__1.MoveNext","level":13,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":14,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":15,"line":0},{"assembly":"System.Net.WebSockets, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.WebSockets.ManagedWebSocket+<EnsureBufferContainsAsync>d__75.MoveNext","level":16,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":17,"line":0},{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":18,"line":0},{"assembly":"System.Net.WebSockets, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Net.WebSockets.ManagedWebSocket+<ReceiveAsyncPrivate>d__652.MoveNext","level":19,"line":0}],"outerId":"3247361","message":"Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.","severityLevel":"Error","type":"System.IO.IOException","id":"46617528"},{"outerId":"46617528","message":"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond","severityLevel":"Error","type":"System.Net.Sockets.SocketException","id":"53239919"}]1 task, Int32 min, Int32 initial)\r\n at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory1 buffer)\r\n at System.Net.Http.HttpConnection.ReadBufferedAsyncCore(Memory1 destination)\r\n at System.Net.Http.HttpConnection.RawConnectionStream.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)\r\n at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)\r\n at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)\r\n at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)\r\n at Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.StartReceiving(WebSocket socket)\r\n at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()\r\n at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)\r\n at System.IO.Pipelines.Pipe.GetReadAsyncResult()\r\n at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)\r\n at Microsoft.Azure.SignalR.ServiceConnectionBase.ProcessIncomingAsync()","ServiceConnectionId":"8f916f18-2d11-496a-bdff-a9dad81eb434"}And then a few dozen of this error I'm assuming one for each connected client:
The text was updated successfully, but these errors were encountered: