"dotnet build-server shutdown" warning for missing dll

[333fred]

Discussed in #74429

^{Originally posted by pjanotti July 17, 2024}
It seems that a recent upgrade of the runtime on GH runners started to trigger a warning and non-zero exit code for the dotnet build-server shutdown command executed from tests in our repo. This seems to happen with runtime 8.0.7, but, it is also not 100% consistent. sometimes it passes in a retry - the test checks for the exit code of the command. It looks like a bug that should be fixed. Could you please confirm if that is the case?

  13:16:22 [DBG]  Running: dotnet build-server shutdown
  13:16:22 [DBG]  ProcessId: 6839
  13:16:22 [DBG]  Exit Code: 1
  13:16:22 [DBG]  StandardOutput:
  13:16:22 [DBG]  Shutting down MSBuild server...
  13:16:22 [DBG]  Shutting down VB/C# compiler server...
  13:16:22 [DBG]  VB/C# compiler server failed to shut down: The shutdown command failed: Error:
  13:16:22 [DBG]    An assembly specified in the application dependencies manifest (VBCSCompiler.deps.json) was not found:
  13:16:22 [DBG]      package: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler', version: '4.10.0-3.24314.14'
  13:16:22 [DBG]      path: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll'
  13:16:22 [DBG]  
  13:16:22 [DBG]  MSBuild server shut down successfully.
  13:16:22 [DBG]  

@Kielek investigated, see here, and

found the Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll file in following locations

C:\Program Files\dotnet\sdk\8.0.302\DotnetTools\dotnet-format
C:\Program Files\dotnet\sdk\8.0.303\DotnetTools\dotnet-watch\8.0.303-servicing.24317.6\tools\net8.0\any
C:\Program Files\dotnet\sdk\8.0.400-preview.0.24324.5\DotnetTools\dotnet-watch\8.0.400-preview.24323.4\tools\net8.0\any\

and what is more important there is no such file in:

C:\Program Files\dotnet\sdk\8.0.303\Roslyn\bincore\Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll
C:\Program Files\dotnet\sdk\8.0.400-preview.0.24324.5\Roslyn\bincore\Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll

When I manually copied the appropriate version locally to these folders, dotnet build-server shutdown executed from unit tests started working. I suppose that this file should be deployed there or reference should be removed.

cc @rajkumar-rangaraj

[333fred]

Related to #73697.

[pjanotti]

One related question: in the project that we detected this we have a temporary work around for dotnet server-build shutdown, but, I noticed that similar issue can happen with dotnet build, but the error is not deterministic on GH runners. For instance, the error below, with dotnet build, succeeded on retry. Could you, please, enlighten me why this failure is not deterministic?

https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/actions/runs/9995580091/job/27628688704?pr=3517#step:9:1303

  17:26:47 [DBG]  Running: dotnet build
  17:26:47 [DBG]  ProcessId: 12914
  17:26:47 [DBG]  Exit Code: 1
  17:26:47 [DBG]  StandardOutput:
  17:26:47 [DBG]    Determining projects to restore...
  17:26:47 [DBG]    All projects are up-to-date for restore.
  Error:  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error : Error: [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  17:26:47 [ERR]  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error : Error: [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  Error:  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :   An assembly specified in the application dependencies manifest (csc.deps.json) was not found: [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  17:26:47 [ERR]  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :   An assembly specified in the application dependencies manifest (csc.deps.json) was not found: [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  Error:  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :     package: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler', version: '4.10.0-3.24314.14' [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  17:26:47 [ERR]  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :     package: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler', version: '4.10.0-3.24314.14' [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  Error:  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :     path: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll' [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  17:26:47 [ERR]  /usr/share/dotnet/sdk/8.0.303/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :     path: 'Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler.dll' [/tmp/otel-dotnet-test-da885c32d7f643f68d297336058aacd9/OTelDotNetCliTest/OTelDotNetCliTest.csproj]
  17:26:47 [DBG]  
  17:26:47 [DBG]  Build FAILED.
  17:26:47 [DBG]  

[333fred]

Going to keep this open for now to track whether we need to backport.

@pjanotti we're not entirely certain why this wouldn't be deterministic.

[pjanotti]

@333fred we put a workaround in place, but, from time to time we still hit this on GH runner. Any idea or info that we could collect to get to the bottom of it?

Our workaround https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/pull/3520/files clearly is not good enough, it can fail with different messages, but, I don't understand why it fail sometimes in one step and some times not.

Two recent instances of hitting this (typically succeeds on first retry):

1. https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/actions/runs/10271690122/job/28423008346#step:9:1336
2. Bump actions/upload-artifact from 4.3.4 to 4.3.5 open-telemetry/opentelemetry-dotnet-instrumentation#3554 (comment)

[333fred]

What version of .NET are you using here?

[pjanotti]

We install via GH action:

Run actions/setup-dotnet@6bd8b7f7774af54e05809fcc5431931b3eb1ddee
  with:
    dotnet-version: 6.0.424
  7.0.410
  8.0.303
  
    cache: false
  env:
    NUGET_PACKAGES: /home/runner/work/opentelemetry-dotnet-instrumentation/opentelemetry-dotnet-instrumentation/packages
    DOTNET_CLI_TELEMETRY_OPTOUT: 1
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --runtime dotnet --channel LTS
dotnet-install: .NET Core Runtime with version '8.0.7' is already installed.
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --version 6.0.424
dotnet-install: .NET Core SDK with version '6.0.424' is already installed.
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --runtime dotnet --channel LTS
dotnet-install: .NET Core Runtime with version '8.0.7' is already installed.
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --version 7.0.410
dotnet-install: .NET Core SDK with version '7.0.410' is already installed.
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --runtime dotnet --channel LTS
dotnet-install: .NET Core Runtime with version '8.0.7' is already installed.
/home/runner/work/_actions/actions/setup-dotnet/6bd8b7f7774af54e05809fcc5431931b3eb1ddee/externals/install-dotnet.sh --skip-non-versioned-files --version 8.0.303
dotnet-install: .NET Core SDK with version '8.0.303' is already installed.

[333fred]

The bugfix was done in .NET 9. @jaredpar for consideration on whether we should backport this fix.

[Kielek]

@333fred. @jaredpar, any chance to backport to .NET8? This version will be longer with as than .NET9.