-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Application not closing socket stuck in CLOSE_WAIT state #36293
Comments
Tagging subscribers to this area: @dotnet/ncl |
@jogoertzen-stantec does it use high CPU for calling |
@karelz It sure does! 😅 Is this a known issue? We unfortunately cannot upgrade to 3.1 at this time as odata is not supported. |
Do you have small code to repro this? I don't think we have enough here to diagnose the problem -- at first glance it appears someone is calling edit: actually, I missed your stack trace. This appears to be something in SqlClient:
|
I think it is essentially dup of #29327 |
BTW some background, we could leave OS descriptor open if it is not disposed explicitly. |
Given the presence of while (readBytes < TdsEnums.HEADER_LEN)
{
readBytes += async ?
await _stream.ReadAsync(packetData, readBytes, TdsEnums.HEADER_LEN - readBytes, token).ConfigureAwait(false) :
_stream.Read(packetData, readBytes, TdsEnums.HEADER_LEN - readBytes);
} Note that the readBytes value is not checked for 0 so you can sit in a non-iterating infinite loop if the connection is somehow closed but not throwing on access. In Microsoft.Data.SqlClient I've got an open PR that changes that behaviour, dotnet/SqlClient#541 |
So, we have a few suspicion. Is there any easy way to confirm which one it is? |
If the runtime fix for later versions makes it impossible to return 0 on a closed socket that'll fix it for most people. In this particular case my upcoming change in Microsoft.Data.SqlClient will fix it on the older runtime. So I don't think there's anything that needs to be done here. My suggestion would be that @jogoertzen-stantec probably needs to track the PR I linked and once it gets merged try the Microsoft.Data.SqlClient preview that contains it to see if the issue goes away. |
@Wraith2 Unless I am missing something, the potential change to |
Yes. |
Great! I am subscribed to dotnet/SqlClient#541 as @Wraith2 suggested. We are planning to upgrade the application to ASP.NET Core 3.1 anyways in the coming weeks, but in case that plan falls through this is a nice alternative. Also, thank you all very much for jumping on this issue so deliciously fast. 😄 Honestly, I wasn't even sure I was in the right place to begin with, so all this attention is greatly appreciated! |
Actually, should I close this issue at this point, or should it remain open until I can confirm one of the proposed approaches actually solves the issue? |
Given that it is not actionable now (#36293 (comment)), we can close it. |
Our ASP.NET Core application (RHEL 7; ASP.NET Core 2.2.7 runtime) frequently stops responding immediately after restarting a remote SQL Server database hosted on Windows Server.
The application has a thread that is repeatedly calling recvmsg on a socket in CLOSE_WAIT state.
The socket was originally ESTABLISHED with the database and transitioned to CLOSE_WAIT as soon as the database host was restarted. The socket remains in CLOSE_WAIT indefinitely until the application is restarted.
Attaching a debugger reveals the following stack trace.
stack.txt
I'm not exactly sure, but this is my best guess as to the line of code that produces the recvmsg calls.
If the TCP state transition diagram below is to be believed, then the socket correctly transitioned to CLOSE_WAIT when the database went down, but the application seems to think it is still ESTABLISHED or something.
Given the information above, does this look like a dotnet bug?
Thanks for reading!
The text was updated successfully, but these errors were encountered: