Runtime Tests Linux_musl arm fails with "The "FindDotNetCliPackage" task failed unexpectedly."

[sandreenko]

Full log:

  Starting Azure Pipelines Test Run CoreCLR Linux arm Checked no_tiered_compilation @ (Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: The "FindDotNetCliPackage" task failed unexpectedly. [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found). [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.ExecuteAsync() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 61 [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 45 [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
##[error].packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Helix) The "FindDotNetCliPackage" task failed unexpectedly.
System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.ExecuteAsync() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 61
   at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 45
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()

An example:
https://dev.azure.com/dnceng/public/_build/results?buildId=882549&view=logs&j=22eeac8c-742e-5aad-c88c-3907a882a5f0&t=8e36944f-99e4-5f3f-4391-f6a5a224ec79

It looks like the last item that blocks at least some green runs after we disabled individual failing tests, should we disable the job?

Link #702

Runfo Tracking Issue: Linux Musl ARM Outerloop can't find dotnet cli package

Definition	Build	Kind	Job Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[sandreenko]

PTAL @tommcdon @trylek @dotnet/runtime-infrastructure

[hoyosjs]

Turning this into a tracking issue. It seems to only affect linux-musl-arm outerloop pipelines, but there's 106 hits over 7 days, so it seems somewhat deterministic.

[trylek]

Hmm, that's weird. @janvorli, it looks like we're actually running the linux_musl arm tests in outerloop runs only, not in PR / CI runs. Did you try to run the outerloop pipeline when introducing the leg i.o.w. do we know if this ever worked?

[safern]

It seems like it is trying to download it from: https://dotnetcli.azureedge.net/dotnet/Runtime/5.0.0-rc.2.20475.5/dotnet-runtime-5.0.0-rc.2.20475.5-linux-musl-arm.tar.gz

I wonder if we shipped a runtime for linux-musl-arm for that version of the runtime.

[janvorli]

We are running the libraries linux_musl_arm tests in the CI. I can see the leg in the recent CIs. It seems that run is executed only if coreclr was changed.

[janvorli]

Here is an example of a log from such a test run:
https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/872722/logs/1053

[safern]

This works: https://dotnetcli.azureedge.net/dotnet/Runtime/5.0.0/dotnet-runtime-5.0.0-linux-musl-arm.tar.gz

It seems like we're using the BundledNetCoreAppVersion which is defined by the sdk used to build... in this case 5.0.0-rc.2.*

I suggest changing this:

runtime/src/tests/Common/helixpublishwitharcade.proj

Line 97 in b37f10a

<DotNetCliVersion>$(BundledNETCoreAppPackageVersion)</DotNetCliVersion>

to 5.0.0 in the meantime until we update our SDK to use 5.0.0 in our global.json

[safern]

Btw, @janvorli, it seems like Alpine ARM is not included in the release notes for 5.0.0: https://github.com/dotnet/core/blob/master/release-notes/5.0/5.0.0/5.0.0.md#net-500---november-10-2020

[trylek]

@janvorli - I believe the log you mention corresponds to a "libraries" Linux_musl arm run, the run that's failing in the outerloop pipeline is a "coreclr" Linux_musl arm run; these two still use somewhat separate scripts and so the properties are likely forked, i.o.w. we should probably do what Santi suggests for the time being.

[janvorli]

That's what I said - we are running only the libraries tests in the CI.

[trylek]

I see, sorry about missing that bit in your previous response.

[janvorli]

it seems like Alpine ARM is not included in the release notes for 5.0.0:

It is intentional, I have found only last week that the aspnetcore packages are not built for 5.0. I have thought I've seen 5.0-RC2 ones being produced after I've merged in my change to aspnetcore, but I've obviously made a mistake and seen something else. So we will fix that for 5.0.1.

[safern]

Sounds good. Didn't know if it was intentional or missed. Thanks for the update.

[ViktorHofer]

Closing as not observable anymore.