HTTP Error 500.30 - ANCM In-Process Start Failure

[ZonerZoner]

Describe the bug

Migrated .NET Core 2.2 application to .NET Core 3.1 does not launch if hostingModel="inprocess" is set in web.config. If the hostingModel="inprocess" is removed, the application loads as expected (in inprocess mode).

Expected behavior

Application launches also with hostingModel="inprocess" set

Additional context
eventlog:
Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication' hit unexpected managed exception, exception code = '0xe0434352'. Please check the stderr logs for more information.
Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication' failed to load clr and
managed application. CLR worker thread exited prematurely

stdout: blank file

additional debug:

[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 10508.. File Version: 13.1.19320.0. Description: IIS ASP.NET Core Module V2. Commit: 2b7e994b8a304700a09617ffc5052f0d943bbcba.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\Web.Backend.dll' path: 'C:\inetpub\myapplication\'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2.dll] Resolving absolute path to dotnet.exe from 'dotnet.exe'
[aspnetcorev2.dll] Invoking where.exe to find dotnet.exe
[aspnetcorev2.dll] where.exe invocation returned: 'C:\Program Files\dotnet\dotnet.exe
C:\Program Files (x86)\dotnet\dotnet.exe
'
[aspnetcorev2.dll] Current process bitness type detected as isX64=1
[aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Binary type 6
[aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\3.1.0\hostfxr.dll'
[aspnetcorev2.dll] Converted argument '.\Web.Backend.dll' to 'C:\inetpub\myapplication\.\Web.Backend.dll'
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\3.1.0\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Argument[1] = 'C:\inetpub\myapplication\.\Web.Backend.dll'
[aspnetcorev2.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\3.1.0\hostfxr.dll
[aspnetcorev2.dll] Canceling standard stream pipe reader
[aspnetcorev2.dll] Loading request handler:  'C:\inetpub\myapplication\runtimes\win-x64\nativeassets\netcoreapp2.2\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'C:\inetpub\myapplication\runtimes\win-x64\nativeassets\netcoreapp2.2\aspnetcorev2_inprocess.dll'. Process Id: 10508.. File Version: 12.2.18316.0. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: ce8cf65589734f82b0536c543aba5bd60d0a5a98.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\Web.Backend.dll' path: 'C:\inetpub\myapplication\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\3.1.0\hostfxr.dll'
[aspnetcorev2_inprocess.dll] Converted argument '.\Web.Backend.dll' to 'C:\inetpub\myapplication\.\Web.Backend.dll'
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\3.1.0\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Argument[1] = 'C:\inetpub\myapplication\.\Web.Backend.dll'
[aspnetcorev2_inprocess.dll] Failed HRESULT returned: 0x800700b7 at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:415 
[aspnetcorev2_inprocess.dll] Throwing ResultException for HRESULT 0x800700b7 at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:197 
[aspnetcorev2_inprocess.dll] Event Log: 'Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication\' failed to load clr and managed application. Unexpected exception: HRESULT 0x800700b7 returned at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:197 ' 
End Event Log Message.
[aspnetcorev2_inprocess.dll] Exception 'HRESULT 0x800700b7 returned at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:197 ' caught at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:302 
[aspnetcorev2_inprocess.dll] Stopping in-process worker thread
[aspnetcorev2_inprocess.dll] Stopping CLR
[aspnetcorev2_inprocess.dll] Event Log: 'Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication\' failed to load clr and managed application. CLR worker thread exited prematurely' 
End Event Log Message.
[aspnetcorev2_inprocess.dll] Exception 'InvalidOperationException' caught at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\inprocessapplication.cpp:350 
[aspnetcorev2_inprocess.dll] Failed HRESULT returned: 0x8007023e at c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\inprocessrequesthandler\dllmain.cpp:120 
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\myapplication\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread

[corchi72]

I also had the same problem as you, I used a standard library from an aspnetcore 3.0-3.1 project.
I had not noticed that in the Standard 2.1 library I had left the Microsoft.AspNetCore = 2.2 library, I commented on the line and then removed the library and the problem was solved !!

<Project Sdk = "Microsoft.NET.Sdk">

   <PropertyGroup>
     <TargetFramework> netstandard2.1 </ targetFramework>
   </ PropertyGroup>

   <ItemGroup>
     <! - <PackageReference Include = "Microsoft.AspNetCore" Version = "2.2.0" /> ->
     <PackageReference Include = "Microsoft.AspNetCore.Mvc" Version = "2.2.0" />

[victormarante]

We are having the same issue @ZonerZoner mentioned. Removing hostingModel="inprocess" fixes the issue for us. We are also making sure only 3.x packages are referenced, so that's not an issue.

Is there a way to set this value directly from release/deploy in Azure DevOps? It's not really feasible to go in and change this every time we redeploy our App Services.

[analogrelay]

@ZonerZoner @victormarante this looks like #18334. Can you look at the workarounds there and try applying them? If they work, the issue should be fixed in an upcoming patch. If not, let us know and it might be something different.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.

[ZonerZoner]

In may case, I could solve the issue. There was a netstandard2.0 library down the road which relied on AspNetCore.App 2.0. After I migrated the library to netcoreapp3.1 (and its dependencies), the issue was gone. But still the fun fact is that the application loaded if the web.config was modified.

[analogrelay]

So you had mixed ASP.NET Core 2.0 and 3.1 dependencies in your app? Given that, it's expected that you'd hit start failures like this since that's not really supported.

[ZonerZoner]

Yes, it was mixed by mistake (in dependent libraries). But the fun fact is that only the web.config attribute is crashing the startup in IIS. If this could be fixed, it would be easier to migrate, I guess

[analogrelay]

If the hostingModel="inprocess" is removed, the application loads as expected (in inprocess mode).

This seems extremely odd. @ZonerZoner can you post a runnable sample reproducing that problem? It should absolutely not be running in-process if you remove that attribute, it should run in out-of-process mode instead.

Can you clarify what (if any) remaining action we have here? I'd like to get to the bottom of the hostingModel attribute issue you mentioned above, but it sounds like the main issue (start failure) has been resolved by fixing your dependencies, correct?

[jkotalik]

So by removing the attribute from the web.config, you are actually running in out-of-process mode, rather than in-process.

Regarding:

Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication' hit unexpected managed exception, exception code = '0xe0434352'. Please check the stderr logs for more information.
Application '/LM/W3SVC/2/ROOT' with physical root 'C:\inetpub\myapplication' failed to load clr and
managed application. CLR worker thread exited prematurely

Do you happen to see any managed logs from ANCM when enabling stdoutLogging? See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-3.1#configuration-with-webconfig

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.

[ZonerZoner]

can you post a runnable sample reproducing that problem?

I guess it's no longer necessary.

but it sounds like the main issue (start failure) has been resolved by fixing your dependencies, correct?

Yes, correct.

So by removing the attribute from the web.config, you are actually running in out-of-process mode, rather than in-process.

I just deployed an old version of the application in our test system and it looks like your argument is valid. I see a w3wp and dotnet process. I would have bet to only have seen the w3wp process before. I guess this issue is stale then.

Do you happen to see any managed logs from ANCM when enabling stdoutLogging?

No. I never saw any logs there as written in the initial post (stdout: blank file)

[analogrelay]

Ok, we'll close this then as it sounds like there isn't much to move forward on here. Feel free to file a new one if something else comes up.