Test failures: System.Net.Http.Functional.Tests.HttpClientHandler_* / * - Azure endpoints

[ghost]

Multiple tests fail with the same stack indicating that the request took more than the configured (default=100s) timeout for HttpClient.
The reasons for this timeout may be different but in most cases the way the tests report the error makes it difficult (or impossible) to understand why the server didn't reply (or why the client didn't receive that reply).

• System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test/NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds
• System.Net.Http.Functional.Tests.HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds(acceptedProtocol: Tls11, requestOnlyThisProtocol: False)
• System.Net.Http.Functional.Tests.HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds(acceptedProtocol: Tls11, requestOnlyThisProtocol: True)
• System.Net.Http.Functional.Tests.HttpClientHandlerTest/GetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect(statusCode: 200)
• System.Net.Http.Functional.Tests.HttpClientHandlerTest/Proxy_BypassFalse_GetRequestGoesThroughCustomProxy(creds: null, wrapCredsInCache: False)
• System.Net.Http.Functional.Tests.HttpClientTest.SendAsync_ExpectedDiagnosticSourceLogging
• System.Net.Http.Functional.Tests.PostScenarioTest.PostUsingChunkedEncoding_Success(serverUri: http://corefx-net.cloudapp.net/Echo.ashx)
• System.Net.Http.Functional.Tests.PostScenarioTest/PostEmptyContentUsingContentLengthSemantics_Success(serverUri: http://corefx-net.cloudapp.net/Echo.ashx)
• System.Net.Http.Functional.Tests.PostScenarioTest.PostUsingUsingConflictingSemantics_UsesChunkedSemantics
• System.Net.Http.Functional.Tests.ResponseStreamTest/ReadAsStreamAsync_Cancel_TaskIsCanceled

OS Distribution:

• (most likely to fail) Windows.7.Amd64-x86 Debug
• Windows.10.Nano.Amd64-x64 Release
• Windows.10.Amd64-x64 Release
• Windows.10.Amd64-x86 Release
• Windows.7.Amd64-x86 Release

---

Opened on behalf of @JiayiLi1

The test System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test/NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds has failed.

System.Net.Http.HttpRequestException : An error occurred while sending the request.\r
---- System.Net.Http.WinHttpException : The operation has been canceled

    Stack Trace:

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test.<NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
----- Inner Stack Trace -----
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()

Build : Master - 20170512.01 (Core Tests)
Failing configurations:

• Windows.10.Amd64-x86
  • Release
• Windows.81.Amd64-x86
  • Release

Detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170512.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FNoCallback_RevokedCertificate_NoRevocationChecking_Succeeds

[karelz]

Dupe in dotnet/corefx#19704 on 2017/5/12

Opened on behalf of @JiayiLi1

The test System.Net.Http.Functional.Tests.HttpClientHandlerTest/GetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect(statusCode: 200) has failed.

System.Net.Http.HttpRequestException : An error occurred while sending the request.\r
---- System.Net.Http.WinHttpException : The operation has been canceled

    Stack Trace:

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest.<>c__DisplayClass46_1.<<GetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Test.Common.LoopbackServer.<>c__DisplayClass3_0.<CreateServerAsync>b__0(Task t)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest.<>c__DisplayClass46_0.<<GetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Test.Common.LoopbackServer.<>c__DisplayClass3_0.<CreateServerAsync>b__0(Task t)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandlerTest.<GetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
----- Inner Stack Trace -----
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()

Build : Master - 20170512.01 (Portable Core Tests)
Failing configurations:

• Windows.10.Nano.Amd64-x64
  • Release

Detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170512.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandlerTest~2FGetAsync_AllowAutoRedirectTrue_NonRedirectStatusCode_LocationHeader_NoRedirect(statusCode:%20200)

[karelz]

@CIPop can you please add all other dupes here?

[CIPop]

again

@danmosemsft That failure is different - please open a different bug for RedHat and that particular error:
System.Net.Http.HttpRequestException : An error occurred while sending the request.\n---- System.Net.Http.CurlException : Problem with the SSL CA cert (path? access rights?)

[danmoseley]

@CIPop will do, deleted my comment.

[karelz]

There is plenty of dupes in just 1 week.
We should treat this one as P0 and consider porting it to 2.0 branch if we track it down in time.

@DavidGoll @CIPop @davidsh any pushback?

[davidsh]

Getting this error:

System.Net.Http.WinHttpException : The operation has been canceled

is impossible to diagnose unless we get a consistent repro. It can be caused by the Internet and/or test servers (on Azure) being slow to respond. It can also be caused by CI machines too busy to process a network I/O.

While there are tests that run into this problem, calling a P0 isn't actionable right now unless we get a repro.

[karelz]

This is P0 due to its impact on our ability to do business - it affects all CoreFX repo developers, all contributors. We need to work on it as P0-ish thing. That's an actionable thing.
We need to somehow address it. If it means disabling tests, fixing tests or fixing product, that's a decision I expect the area owners to make.
Trying to get a repro may be part of the job, if the area owners decide that's the right approach.

The bottom line is, that we can't live with flaky tests and keep doing further progress on the product.
Makes sense?

[steveharter]

Note that GetAsync_AllowedSSLVersion_Succeeds repros on Win7 CI machines at a ~3% failure rate (see dup'd issue https://github.com/dotnet/corefx/issues/13281)

[KristinXie1]

This issue is repro on Portable Core Tests, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170522.01/workItem/System.Net.Http.Functional.Tests/wilogs

[danmoseley]

also https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170522.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandlerTest~2FProxy_BypassFalse_GetRequestGoesThroughCustomProxy(creds:%20null,%20wrapCredsInCache:%20False)

System.AggregateException : One or more errors occurred. (Object reference not set to an instance of an object.) (An error occurred while sending the request.) (One or more errors occurred. (An error occurred while sending the request.))\r\n
---- System.NullReferenceException : Object reference not set to an instance of an object.\r\n
---- System.Net.Http.HttpRequestException : An error occurred while sending the request.\r\n
-------- System.Net.Http.WinHttpException : The operation has been canceled\r\n
---- System.AggregateException : One or more errors occurred. (An error occurred while sending the request.)\r\n
-------- System.Net.Http.HttpRequestException : An error occurred while sending the request.\r\n
------------ System.Net.Http.WinHttpException : The operation has been canceled

NullReferenceException doesn't seem right, even in the face of a networking error. But it seems it may be from tests/xunit:

----- Inner Stack Trace dotnet/corefx#1 (System.NullReferenceException) -----
   at System.Net.Http.Functional.Tests.LoopbackGetRequestHttpProxy.<>c__DisplayClass2_0.<<StartAsync>b__0>d.MoveNext()

[CIPop]

To make progress we could instrument the test with a Debug.Assert/Fail when this exception is raised and obtain a full-memory dump to look at the server-side.

[KristinXie1]

Failed again: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170621.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FNoCallback_RevokedCertificate_NoRevocationChecking_Succeeds

[v-haren]

failed again: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_win7_debug/90/testReport/System.Net.Http.Functional.Tests/ResponseStreamTest/ReadAsStreamAsync_ValidServerResponse_Success_transferType__Chunked__transferError__None_/

[KristinXie1]

failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170822.01/workItem/System.Net.Http.Functional.Tests/analysis/xunit/System.Net.Http.Functional.Tests.HttpClientHandler_ServerCertificates_Test~2FUseCallback_NotSecureConnection_CallbackNotCalled

[KristinXie1]

Following test: System.Net.Http.Functional.Tests.HttpClientHandler_MaxConnectionsPerServer_Test.GetAsync_MaxLimited_ConcurrentCallsStillSucceed failed with same issue, detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_win7_debug/271/testReport/System.Net.Http.Functional.Tests/HttpClientHandler_MaxConnectionsPerServer_Test/GetAsync_MaxLimited_ConcurrentCallsStillSucceed_maxConnections__3__numRequests__2__secure__True_/

MESSAGE:

System.Net.Http.HttpRequestException : An error occurred while sending the request.\r\n---- System.Net.Http.WinHttpException : The operation has been canceled

STACK TRACE:

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() 
at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext() in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 464 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() 
at System.Net.Http.Functional.Tests.HttpClientHandler_MaxConnectionsPerServer_Test.<GetAsync_MaxLimited_ConcurrentCallsStillSucceed>d__3.MoveNext() in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.MaxConnectionsPerServer.cs:line 74 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
----- Inner Stack Trace ----- 
at System.Environment.get_StackTrace() in D:\j\workspace\outerloop_net---92764178\src\System.Runtime.Extensions\src\System\Environment.cs:line 174 
at System.Runtime.ExceptionServices.ExceptionStackTrace.AddCurrentStack(Exception exception) in D:\j\workspace\outerloop_net---92764178\src\Common\src\System\Runtime\ExceptionServices\ExceptionStackTrace.cs:line 23 
at System.Net.Http.WinHttpException.CreateExceptionUsingError(Int32 error) in D:\j\workspace\outerloop_net---92764178\src\Common\src\System\Net\Http\WinHttpException.cs:line 56 
at System.Net.Http.WinHttpRequestCallback.OnRequestError(WinHttpRequestState state, WINHTTP_ASYNC_RESULT asyncResult) in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 324 
at System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr handle, WinHttpRequestState state, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength) in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 104 
at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr handle, IntPtr context, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength) in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs:line 47 
--- End of stack trace from AddCurrentStack --- 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() 
at System.Threading.Tasks.RendezvousAwaitable`1.GetResult() in D:\j\workspace\outerloop_net---92764178\src\Common\src\System\Threading\Tasks\RendezvousAwaitable.cs:line 62 
at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext() in D:\j\workspace\outerloop_net---92764178\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpHandler.cs:line 856

[Sunny-pu]

failed again in CI:
https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_windows_nt_release/209/testReport/System.Net.Http.WinHttpHandlerFunctional.Tests/ServerCertificateTest/NoCallback_ValidCertificate_CallbackNotCalled/

[wfurt]

It did not failed sine June. The non-managed failures were tracked here https://github.com/dotnet/corefx/issues/13281
Please re-open if this fails again.

[karelz]

This is tracking unreliable Azure infrastructure (Azure end points). Reopening.
E.g. test PostAsync_CallMethod_StreamContent (closed as dupe) failed in Sep, Oct and Nov.

[rmkerr]

I'm getting very consistent timeouts for PostAsync_CallMethod_StreamContent with the same "System.Net.Http.WinHttpException : The operation has been canceled" timeout that is showing up for the rest of the test failures mentioned here. If you think it would help I can collect wireshark traces of the failure and share them.

        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Http.WinHttpException : The operation has been canceled
        Stack Trace:
           C:\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs(464,0): at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
           c:\corefx\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.cs(1463,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest.<PostAsync_CallMethod_StreamContent>d__80.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
           --- End of stack trace from previous location where exception was thrown ---
           --- End of stack trace from previous location where exception was thrown ---
           ----- Inner Stack Trace -----
           C:\corefx\src\System.Runtime.Extensions\src\System\Environment.cs(176,0): at System.Environment.get_StackTrace()
           C:\corefx\src\Common\src\System\Runtime\ExceptionServices\ExceptionStackTrace.cs(23,0): at System.Runtime.ExceptionServices.ExceptionStackTrace.AddCurrentStack(Exception exception)
           C:\corefx\src\Common\src\System\Net\Http\WinHttpException.cs(56,0): at System.Net.Http.WinHttpException.CreateExceptionUsingError(Int32 error)
           C:\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs(324,0): at System.Net.Http.WinHttpRequestCallback.OnRequestError(WinHttpRequestState state, WINHTTP_ASYNC_RESULT asyncResult)
           C:\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs(104,0): at System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr handle, WinHttpRequestState state, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
           C:\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpRequestCallback.cs(47,0): at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr handle, IntPtr context, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
           --- End of stack trace from AddCurrentStack ---
           C:\corefx\src\Common\src\System\Threading\Tasks\RendezvousAwaitable.cs(62,0): at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
           C:\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpHandler.cs(856,0): at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()```

[karelz]

That one is tracked primarily by dotnet/corefx#25902 and there's bunch of other tests with similar exception/error sets.
If you can reproduce, we should try to collect additional logs to find out what's going on.

[karelz]

Triage: We don't seem to be getting much value from this particular tracking issue. We're on top of Networking failures and there are many specific tracking bugs for the common failures. Closing.