Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline Caching failures should not fail the build #10917

Closed
lukeapage opened this issue Jul 17, 2019 · 12 comments
Closed

Pipeline Caching failures should not fail the build #10917

lukeapage opened this issue Jul 17, 2019 · 12 comments

Comments

@lukeapage
Copy link

lukeapage commented Jul 17, 2019

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: CacheBeta

Environment

  • Server - Azure Pipelines or TFS on-premises?

    • If using TFS on-premises, provide the version:

    • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

  • Agent - Hosted or Private:

    • If using Hosted agent, provide agent queue name:

    • If using private agent, provide the OS of the machine running the agent and the agent version:

Issue Description

When we get a failure on the pipeline which has happened quite a lot recently, it should use save that the cache was not restored (and not try to save it again).

Task logs

Information, Downloaded 789.2 MB out of 817.4 MB (97%).
Information, Response exception thrown: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown.
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpMethod method, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.GetDedupUrlsRawAsync(ISet`1 dedupIds, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.GetDedupGettersAsync(ISet`1 dedupIds, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.<>c__DisplayClass26_1.<<DownloadToFileAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.DownloadToFileAsync(DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupIdentifier dedupId, String fullPath, UInt64 fileSize, GetDedupAsyncFunc dedupFetcher, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.<>c__DisplayClass24_0.<<DownloadAsyncWithManifestPath>b__6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.TargetBlockExtensions.SendAllAndCompleteAsync[T1,T2](ITargetBlock`1 targetBlock, IEnumerable`1 inputs, ITargetBlock`1 finalBlock, CancellationToken token)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.DownloadAsyncWithManifestPath(DownloadDedupManifestArtifactOptions downloadOptions, IEnumerable`1 minimatcherFuncs, Boolean downloadManifestReferences, ISet`1 excludedPaths, CancellationToken cancellationToken) 
 Status code: ServiceUnavailable 
 Inner exception:  
 Source: Microsoft.VisualStudio.Services.WebApi
##[error]Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown.
@ghost ghost added bug route labels Jul 17, 2019
@lukeapage
Copy link
Author

I'm hoping the workaround of using continueOnError: true on the task will solve these issues.

If you think that's the general solution, I recommend adding it to the examples in the docs.

@willsmythe
Copy link
Contributor

Thanks for reporting. We need to investigate why you are getting a ServiceUnavailable error.

Also linking to #10907 since we should also consider replicating the step's continueOnError value (default is false) to the post job (save) step.

@willsmythe willsmythe changed the title Cache failures cause job to fail Cache failures cause job to fail / ServiceUnavailable error during cache download Jul 17, 2019
@lukeapage
Copy link
Author

It was a one off error affecting 2 builds during the same time period. I can give you more details if it would help?

@johnterickson
Copy link
Contributor

Earlier in the logs you should see a line like this:

 DedupManifestArtifactClient will correlate http requests with X-TFS-Session ${GUID}

If you can share your session GUID we can look directly at the telemetry.

@lukeapage
Copy link
Author

Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session a39b617b-566f-4c42-a5d2-ad03bb0994be

It happened a lot today at various times.

It also happened once on a build pipeline artifact (same error)

@lukeapage
Copy link
Author

@willsmythe @johnterickson any progress on why it happens?

I've switched to 7z compressing node_modules before caching which obv. reduces the number of files massively and takes the payload from 800mb to 200mb and that seems to have stopped the errors occurring. I had one build that hadn't yet rebased on the new pipeline and that still failed, heres the logs

Information, There is a cache hit.
Manifest ID is: BC113A87408DC5914B7FBB298FB03D247100CED15F235AA00DD1CE4987EFEC6D02
Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 0972f88e-2037-49b2-898e-2bc81a9835a0
Warning, [https://mzgvsblobprodsu6weus1.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/DFECC24D14563F7BBA5EAF0A0EB703F3F39630A5980BB909809999E1464BD0ED01?sv=2017-04-17&sr=b&sig=MOgTrs7N3FqYMnhTZ26AhaYKJAxsoeNWhfEiOKP6oZc%3D&spr=https&se=2019-07-24T12%3A52%3A28Z&sp=r&rscl=x-e2eid-09b8dfe2-0b6c4786-a2400471-8a1890b7-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Information, Could not initialize dataport.
Information, Downloaded 0.0 MB out of 814.1 MB (0%).
Information, Downloaded 108.7 MB out of 814.1 MB (13%).
Information, Downloaded 148.3 MB out of 814.1 MB (18%).
Information, Downloaded 175.0 MB out of 814.1 MB (21%).
Information, Downloaded 274.3 MB out of 814.1 MB (34%).
Warning, [https://kx9vsblobprodsu6weus66.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/920506ED66C13868153A59AD50D5943AEE9CE0BBDA4429E6C37F0D15712C69C701?sv=2017-04-17&sr=b&sig=bVAdA1al0f2mTyAOun5ERmEk1iCnWRcIxulLYb3UsUc%3D&spr=https&se=2019-07-24T13%3A34%3A17Z&sp=r&rscl=x-e2eid-e1ef265b-1e34448c-9cbedea3-45f1289d-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Information, Downloaded 324.8 MB out of 814.1 MB (40%).
Information, Downloaded 364.2 MB out of 814.1 MB (45%).
Warning, [https://41hvsblobprodsu6weus29.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/BB828D6465B9328A3490487A7742D2D0560E75BD755B8512726356405AE69C7701?sv=2017-04-17&sr=b&sig=0mgoccTgDIdOGPXJCIikVJrLhea4M6kKIES8TqDBxjE%3D&spr=https&se=2019-07-24T12%3A43%3A57Z&sp=r&rscl=x-e2eid-053c6644-2b284f37-808bbc5e-f154124b-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Warning, [https://s83vsblobprodsu6weus72.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/5D5CEBB3CDAE0E76030789B47CD7E4391185069CD086D03DED6849260F1BB2D601?sv=2017-04-17&sr=b&sig=rUkzqgaKXHtoAVTS1oFS3WqYZJfdFreBFg0AThwMMls%3D&spr=https&se=2019-07-24T13%3A21%3A52Z&sp=r&rscl=x-e2eid-053c6644-2b284f37-808bbc5e-f154124b-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Warning, [https://ijgvsblobprodsu6weus39.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/239E65EE14E012943CC24EE31FBAD1E6F512F728FF7A383ECEABE1BD3B42CF5801?sv=2017-04-17&sr=b&sig=LwfrLqZNy9uvAyXcZrvibk5gG1jAqrCnZZH3tn85JsE%3D&spr=https&se=2019-07-24T13%3A08%3A21Z&sp=r&rscl=x-e2eid-2844cb8a-d4c2409a-be695e41-70d89284-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Warning, [https://24uvsblobprodsu6weus18.blob.core.windows.net/dbd2f735ff1e434ce8826b9e3578951332/451A213992BC50FF8D33C605797ABF368C4F43BA42AB6EAA581D29198C011A4601?sv=2017-04-17&sr=b&sig=ippRRRd5LLiaHwkXcJdcDdB6FwgpiILgtFNpDrif55Y%3D&spr=https&se=2019-07-24T13%3A16%3A12Z&sp=r&rscl=x-e2eid-b27b710a-a651458b-974b30f2-3722943b-session-0972f88e-203749b2-898e2bc8-1a9835a0] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Information, Downloaded 381.4 MB out of 814.1 MB (47%).
Information, Downloaded 453.4 MB out of 814.1 MB (56%).
Information, Downloaded 477.4 MB out of 814.1 MB (59%).
Information, Downloaded 535.2 MB out of 814.1 MB (66%).
Information, Downloaded 630.7 MB out of 814.1 MB (77%).
Information, Downloaded 656.0 MB out of 814.1 MB (81%).
Information, Downloaded 748.2 MB out of 814.1 MB (92%).
Information, Downloaded 769.3 MB out of 814.1 MB (95%).
Information, Downloaded 778.7 MB out of 814.1 MB (96%).
Information, Response exception thrown: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown.
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpMethod method, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.GetDedupUrlsRawAsync(ISet`1 dedupIds, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.GetDedupGettersAsync(ISet`1 dedupIds, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.<>c__DisplayClass26_1.<<DownloadToFileAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.DownloadToFileAsync(DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupNode node, String fullPath, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClientWithDataport.DownloadToFileAsync(IDedupDataPort dataport, DedupIdentifier dedupId, String fullPath, UInt64 fileSize, GetDedupAsyncFunc dedupFetcher, Uri proxyUri, EdgeCache edgeCache, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.<>c__DisplayClass24_0.<<DownloadAsyncWithManifestPath>b__6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.NonSwallowingActionBlock.<>c__DisplayClass5_0`1.<<CreateNonSwallowingFunc>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.TargetBlockExtensions.SendAllAndCompleteAsync[T1,T2](ITargetBlock`1 targetBlock, IEnumerable`1 inputs, ITargetBlock`1 finalBlock, CancellationToken token)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.DownloadAsyncWithManifestPath(DownloadDedupManifestArtifactOptions downloadOptions, IEnumerable`1 minimatcherFuncs, Boolean downloadManifestReferences, ISet`1 excludedPaths, CancellationToken cancellationToken) 
 Status code: ServiceUnavailable 
 Inner exception:  
 Source: Microsoft.VisualStudio.Services.WebApi
##[error]Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown.

@johnterickson
Copy link
Contributor

@lukeapage We've increased the capacity of the scale unit you were seeing this on. You should stop seeing this in an hour or two.

@johnterickson
Copy link
Contributor

Capacity increased and throttled no longer observed. Do reactivate if you see this error again.

@lukeapage
Copy link
Author

@johnterickson this issue was also about defaulting continue on error to true for cache tasks, do you want to reopen for that or is it handled elsewhere/ you don’t want to do it?

@johnterickson johnterickson changed the title Cache failures cause job to fail / ServiceUnavailable error during cache download Pipeline Caching failures should not fail the build Jul 24, 2019
@johnterickson
Copy link
Contributor

@willsmythe to confirm, you are support of cache steps being continueOnError?

@johnterickson johnterickson reopened this Jul 24, 2019
@fadnavistanmay
Copy link
Contributor

@lukeapage - We are going to have the users decide whether they want to continue on error of the cache task or not; closing this issue. Thanks.

@viceice
Copy link

viceice commented Oct 16, 2019

Seeing this too on azure hosted agents (system independent)

https://dev.azure.com/renovatebot/renovate/_build/results?buildId=1157

##[section]Starting: Cache Yarn packages
==============================================================================
Task         : Cache (Beta)
Description  : Cache files between runs
Version      : 0.1.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key: yarn_cache|Linux|/home/vsts/work/1/renovate/yarn.lock
 - yarn_cache                           [string]
 - Linux                                [string]
 - /home/vsts/work/1/renovate/yarn.lock [file] --> F0AD86C2B0B74702E6F53D53D79AA09D29B71D38FF030FE378DF2DCDDCF680B5
Resolved to: yarn_cache|Linux|j+ut0SUhVacRYAzw3wqdvJ7lxDA9k5e2XFbaMEIu1sA=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 775be811-ed32-41dc-9145-69d576cba85d
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `yarn_cache|Linux|j+ut0SUhVacRYAzw3wqdvJ7lxDA9k5e2XFbaMEIu1sA=`
Information, There is a cache hit: `yarn_cache|Linux|j+ut0SUhVacRYAzw3wqdvJ7lxDA9k5e2XFbaMEIu1sA=`
Information, Used scope: 1;6ed21661-b561-4777-8106-ca95b548903d;refs/heads/master;renovatebot/renovate
Entry found at fingerprint: `yarn_cache|Linux|j+ut0SUhVacRYAzw3wqdvJ7lxDA9k5e2XFbaMEIu1sA=`
Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodcus3.vsblob.visualstudio.com/A8ee6462d-ab82-4728-954c-a00ca9af968e/_apis/dedup/chunks/80E6E5C2AD755C018635D5FDD543065F10A351DF61528117707AFB936B0D757001 attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False
Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodcus3.vsblob.visualstudio.com/A8ee6462d-ab82-4728-954c-a00ca9af968e/_apis/dedup/nodes/ECCBC8D9FA383C7041BE02D0D1221866B6E5CC600072B496959EEFE5B00C9C6002 attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False
Information, Expected size to be downloaded: 401.2 MB
Information, Downloaded 0.0 MB out of 401.2 MB (0%).
Information, Downloaded 37.4 MB out of 401.2 MB (9%).
Information, Downloaded 73.9 MB out of 401.2 MB (18%).
Information, Downloaded 127.3 MB out of 401.2 MB (32%).
Information, Downloaded 184.9 MB out of 401.2 MB (46%).
Information, Downloaded 266.6 MB out of 401.2 MB (66%).
Information, Downloaded 341.5 MB out of 401.2 MB (85%).
Warning, [https://st8vsblobprodcus339.blob.core.windows.net/db8ee6462dab824728954ca00ca9af968e/5FA2790B529DAFD4F9577CA083E07CFF2C1300C353610FB2C4763A72183B333A01?sv=2017-04-17&sr=b&sig=vPtybVnRWgS5Ve5fKuyOwn69LFWKk4w7p1piPi9bIeg%3D&spr=https&se=2019-10-16T17%3A22%3A25Z&sp=r&rscl=x-e2eid-bbac4036-d1f94a89-9d0a8f52-1e858346-session-775be811-ed3241dc-914569d5-76cba85d] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
No LastRequestResponse on exception HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: SSL connect error
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
Information, ApplicationInsightsTelemetrySender correlated 3 events with X-TFS-Session 775be811-ed32-41dc-9145-69d576cba85d
##[error]A task was canceled.
##[section]Finishing: Cache Yarn packages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants