Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DllNotFoundException after .NET 5 in-place update #60144

Open
TRZLGast opened this issue Oct 7, 2021 · 58 comments
Open

DllNotFoundException after .NET 5 in-place update #60144

TRZLGast opened this issue Oct 7, 2021 · 58 comments
Milestone

Comments

@TRZLGast
Copy link

TRZLGast commented Oct 7, 2021

Description

If Microsoft Update (MU) installs an in-place update of .NET 5 framework (no major/minor feature update) while framework dependent applications are running, then these applications may crash after a time because of missing dlls (FileNotFoundException/DllNotFoundException).

The in-place update installs the new version of the .NET framework and then it tries to delete the old version. If dlls of the old version have been loaded, they can't be deleted and so they were moved to a temporary hidden directory that will be deleted after restart. Moved dlls stay loaded in running applications, but can't be loaded by name/path anymore. And that's exactly the problem.
If one of these applications need to load an additional framework dll, the dotnet loader looks for this dll in the framework directory of the old version. This directory doesn't exist and so the application crashes because of DllNotFound or FileNotFound exception

Steps to reproduce

  1. Create a .NET 5 WPF application with a button. In the click handler call a framework method from an assembly that isn't already loaded (e.g. MessageBox.Show(typeof(System.Printing.PrintPort).Name); )
  2. Start this application on a system with installed .NET framework version 5.0.7
  3. Activate Microsoft Update, search for updates and wait until version 5.0.10 is installed
  4. Click the button of the WPF application
  5. Crash

Application: WpfApp1.exe
CoreCLR Version: 5.0.721.25508
.NET Version: 5.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException: Could not load file or assembly 'System.Printing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

With Procexp you can see the movement of loaded dlls.
Before .NET framework update
image

after .NET framework update
image

With Procmon you can see where the loader is searching for framework dlls after .NET framework update:
image

But this directory doesn't exist anymore.

You can reproduce this problem even easier. After an update of the .NET framework you only need to close the WPF application (no button click or custom code necessary). The application will crash again, but now you doesn't recognize it directly. You have to look into event log:

Application: WpfApp1.exe
CoreCLR Version: 5.0.721.25508
.NET Version: 5.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.DllNotFoundException: Unable to load DLL 'vcruntime140_cor3.dll' or one of its dependencies: Das angegebene Modul wurde nicht gefunden. (0x8007007E)
at __std_type_info_destroy_list(__type_info_node* )
at __scrt_uninitialize_type_info()
at _app_exit_callback()
at .LanguageSupport.DomainUnload(Object A_0, EventArgs A_1)
at .ModuleUninitializer.SingletonDomainUnload(Object source, EventArgs arguments)
at System.AppContext.OnProcessExit()

Because Window Update is running behind the scenes, and the described problems occur much later, it’s not easy to comprehend why the applications are crashing. .NET framework will be updated monthly and so there is a high risk that these types of problems appear.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Oct 7, 2021
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@danmoseley
Copy link
Member

@NikolaMilosavljevic any idea where this issue should go? WPF?

@NikolaMilosavljevic
Copy link
Member

@NikolaMilosavljevic any idea where this issue should go? WPF?

This can stay in runtime as it is something that can be experienced with shared and desktop runtimes.

@danmoseley
Copy link
Member

OK picked area-Setup.

@rkieslinger
Copy link

@danmoseley Is there an actual time table, when we can expect some feedback on this? We are in the middle of the release of our biggest update in years (migration from 4.7.2 to 5.0) and as the first small wave of customers got the update, this issue appeared and it seems huge.

@danmoseley
Copy link
Member

danmoseley commented Oct 11, 2021

@NikolaMilosavljevic @vitek-karas can you help find someone to localize this one? It is blocking @rkieslinger update.

@vitek-karas
Copy link
Member

@joeloff for the installer side of things. I don't think there's much the runtime can do here. If there's a way for the setup to leave the old version in place and only remove it upon next restart - that would probably be the best solution. But I don't know what is the contract with security servicing.

@rkieslinger
Copy link

If I remember correctly, the .NET Runtime update actually did install side by side, but the updates of the .NET Desktop Runtime and ASP.NET Core Runtime did not and so they are the ones that are causing issues.

@TRZLGast can you confirm that, just to be sure?

@joeloff
Copy link
Member

joeloff commented Oct 11, 2021

I don't see much we can do from the setup side either. Upgrades uninstall. There's no way to defer a part of the upgrade until the next restart. Right now we install the new version and remove the old. That would break 99% of the cases where leveraging MU is removing the previously vulnerable pieces from the machine.

We introduced upgrades around 3.0 across the board. There are two models being used.

  1. The shared host upgrades the actual MSI itself
  2. For the standalone installs, runtimes + SDK, the new version of any MSI is laid down first before removing the old one.

This would be less of an issue for .NET Framework because assemblies would end up in the GAC I think

@rkieslinger
Copy link

So to sum it up, we just have these 2 options?

  1. We tell our 20k+ customers, they gotta get used to a random monthly crash and that this is by design?
  2. We disable .net updates via microsoft update on our customers computers (registry key BlockMU). Seems problematic, if there's other software on the computer using .net which is relying on them.

Doesn't MSI sometimes utilize PendingFileRenameOperations to delete files on the next reboot? Wouldn't that be a better solution, instead of snatching runtime files away from running applications?

Another approach which gets to my mind could be an entry in RunOnce to uninstall the old version safely at the next reboot?

@TRZLGast TRZLGast reopened this Oct 12, 2021
@TRZLGast
Copy link
Author

If I remember correctly, the .NET Runtime update actually did install side by side, but the updates of the .NET Desktop Runtime and ASP.NET Core Runtime did not and so they are the ones that are causing issues.

@TRZLGast can you confirm that, just to be sure?

Yes that's right.

before update:
image

after update (and restart):
image

@joeloff
Copy link
Member

joeloff commented Oct 12, 2021

How did you acquire these? The runtime installers will always upgrade, but SDK feature bands are SxS, so if you installed SDK 5.0.100 and then 5.0.101, that will upgrade the runtime from 5.0.0 to 5.0.1.

However, you could then install SDK 5.0.205. the 2xx and 1xx bands are SxS, so that would allow you to have both the 5.0.1 and 5.0.8 runtimes.

If you installed just the runtime, e.g. dotnet-runtime-5.0.7-win-x64.exe and then dotnet-runtime-5.0.8-win-x64.exe, that will upgrade.

But if you installed dotnet-runtime-5.0.7-win-x64.exe and then windowsdesktop-runtime-5.0.7-win-x64.exe followed by dotnet-runtime-5.0.8-win-x64.exe, you'll have the 5.0.7 and 5.0.8 runtimes plus the 5.0.7 desktop runtime. That's because the installers are ref counted against each of the separate bundles (EXEs).

@NikolaMilosavljevic NikolaMilosavljevic removed the untriaged New issue has not been triaged by the area owner label Oct 13, 2021
@NikolaMilosavljevic NikolaMilosavljevic added this to the 7.0.0 milestone Oct 13, 2021
@vitek-karas
Copy link
Member

Adding @dleeapho since this is mostly about runtime installers.

@yyjdelete
Copy link

yyjdelete commented Dec 27, 2021

+1 for this
Not sure, but seems it's worse for run dotnet.exe (instead of the output .exe) refs Microsoft.AspNetCore.App as windows services or website(aspnetcore+IIS).
Seems during upgrade, the installer will kill all dotnet.exe first before installer new one. But for IIS/services in windows, it will restart immediately(So it lost all already loaded dlls and will try to load them latter), but at that time the new framework is not installed, so it still use the old framework while installer begin to remove file of old framework, and random get some FileNotFoundException(maybe catched by framework, so it still running instead of restart again, but the services is broken)

@yyjdelete
Copy link

yyjdelete commented Dec 27, 2021

For upgrade dotnet-hosting OPT_NO_X86 and capture with procmon:

  1. install dotnet-runtime-6.0.1
  2. killall dotnet.exe
    //restart with dotnet6.0.1+aspnetcore6.0.0
  3. uninstall dotnet-runtime-6.0.0
  4. install aspnetcore-runtime-6.0.1
  5. uninstall aspnetcore-runtime-6.0.1

Seems killall dotnet.exe step should be do again at last or just moved to last step, to ensure restarted programs will not do a race between load dlls from framework and remove old frameworks, and only successful to load some of them.

@jairbubbles
Copy link

I've also been seeing this randomly on an app (.NET5 Desktop), it's pretty annoying. I was suspecting that it had to do with .NET update, now I have confirmation.

Keeping the old .NET runtime seems like the best thing to do. I personnally have many runtimes installed in that folder:

image

So I'm not sure why it has to be different with Microsoft Update. We don't want it to grow indefinitely?

@jairbubbles
Copy link

@rkieslinger I guess another option would be to deploy a self-contained app to your customers.

@rkieslinger
Copy link

@rkieslinger I guess another option would be to deploy a self-contained app to your customers.

We evaluated that before migrating to .NET 5.0+, but decided to roll with a shared framework for couple reasons. We'll stick to that decision and implemented some error management so the user at least gets an info that he has to restart his computer.

@jairbubbles
Copy link

As it happens only on .dll not loaded yet (in our case it often happens on automation DLL for instance) I was also thinking to preload them as startup.

@jairbubbles
Copy link

New runtime versions were just released:

image

And we saw similar expections popping and it's clearly not a coincidence.

I was thinking of a new approach by using AssemblyResolve callback. If the requested .dll is not available why not trying to load it from the new version installed? As it's a patch, it's probably safe and assembly versions will match.

Something like:

        public App()
        {
            var runtimeDirectory = System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory().TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);

            AppDomain.CurrentDomain.AssemblyResolve += (_, args) =>
            {
                // Runtime directory is not available anymore, probably deleted by an auto-update, hopefully a new patch should be available
                if (!Directory.Exists(runtimeDirectory) && Version.TryParse(Path.GetFileName(runtimeDirectory), out var runtimeVersion))
                {
                    var basePath = Path.GetDirectoryName(runtimeDirectory)!;

                    // Look for the next runtime patch and load the .dll from there
                    for (int i = 1; i < 10; i++)
                    {
                        var newVersion = new Version(runtimeVersion.Major, runtimeVersion.Minor, runtimeVersion.Build + i);
                        var newAssemblyPath = Path.Combine(basePath, newVersion.ToString(), new AssemblyName(args.Name).Name + ".dll");
                        if (File.Exists(newAssemblyPath))
                        {
                            return Assembly.LoadFrom(newAssemblyPath);
                        }
                    }
                }

                return null;
            };
        }

NB: I'm not sure how Microsoft Updates manage to remove the old folder, if I try to do it manually, explorer will prevent me from doing so as it's being in use.

@jairbubbles
Copy link

It won't work in the current state:

  • There is a twist in the runtime as the runtime path is set to C:\Program Files\dotnet\shared\Microsoft.NETCore.App but the WPF .dll are loaded from C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App
  • Some .dll that fails to load are native .dll and AssemblyResolve, I would need to update the probing path for native too

@TomLeroc
Copy link

I'm seeing a very similar issue, if not the same.

I have encountered this issue with a fully self contained .net core app with WPF elements.
This exception is occurring on the FinalizerThread in response to the application being closed.

pe -lines
Exception object: 0000027a5dcc8490
Exception type: System.DllNotFoundException
Message: Dll was not found.
InnerException:
StackTrace (generated):
SP IP Function
000000F473E7F3C0 0000000000000000 DirectWriteForwarder.dll!.__std_type_info_destroy_list(__type_info_node*)+0x1
000000F473E7F470 0000027A0341F8AA DirectWriteForwarder.dll!.__scrt_uninitialize_type_info()+0x3a
000000F473E7F4B0 0000027A0341DD06 DirectWriteForwarder.dll!.app_exit_callback()+0x116
000000F473E7F590 0000027A0341F320 DirectWriteForwarder.dll!..LanguageSupport.DomainUnload(System.Object, System.EventArgs)+0x30
000000F473E7F5C0 0000027A0342BF5B DirectWriteForwarder.dll!.ModuleUninitializer.SingletonDomainUnload(System.Object, System.EventArgs)+0x9b
000000F473E7F680 00000001801EB4CE System.Private.CoreLib.dll!System.AppContext.OnProcessExit()+0x5e [/
/src/libraries/System.Private.CoreLib/src/System/AppContext.cs @ 81]

StackTraceString:
HResult: 80131524
I have a mini dump, but I don't know whether it is repeatable.

@jairbubbles
Copy link

Looks indeed like it!

Please note that it's hard to reproduce as you need to uninstall .NET while your app is running. Over than that, it seems pretty consistent. Some users reported that it happened with Visual Studio Installer.

I tried a workaround by watching the file changes on the .NET folder and "swap" to the new one but it never kicked in. It looks like that:

// Get WPF runtime directory
var runtimeDirectory = Path.GetDirectoryName(typeof(Window).Assembly.Location);
var sharedRuntimePath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles), "dotnet", "shared", "Microsoft.WindowsDesktop.App");

// Watch parent folder for modifications
if (runtimeDirectory.StartsWith(sharedRuntimePath))
{
    _fileWatcherService.AddWatcher(sharedRuntimePath, OnFolderChangedAsync);
}

Task OnFolderChangedAsync(object sender, FileSystemEventArgs e)
{
    if (!_runtimeDeleted && !Directory.Exists(runtimeDirectory))
    {
        // Runtime was deleted 😲 
        _runtimeDeleted = true;

        // Track exception so that we're aware when all this gets triggered
        _telemetryService.TrackException(new InvalidOperationException(".NET Runtime was deleted"));
        _logger.LogWarning($".NET Runtime was deleted ({runtimeDirectory})");

        // Ask the user to restart praying that he/she clicks on it
        _notificationManager.ShowRestartNotification(".NET Runtime was updated. You need to restart the application.");

        // Try resolving the new framework version
        runtimeDirectory = GetNewRuntimeFolder(runtimeDirectory);
        if (runtimeDirectory == null)
        {
            _logger.LogError(".NET Runtime couldn't be resolved to a new directory.");
            return Task.CompletedTask;
        }
        _logger.LogWarning($".NET Runtime was resolved to new location: {runtimeDirectory}.");

        // Hook up on AssemblyResolve for managed assemblies
        AppDomain.CurrentDomain.AssemblyResolve += (_, args) =>
        {
            var newAssemblyPath = Path.Combine(runtimeDirectory, new AssemblyName(args.Name).Name + ".dll");
            if (File.Exists(newAssemblyPath))
            {
                return Assembly.LoadFrom(newAssemblyPath);
            }
            return null;
        };

        // Update default directories for native .dll
        SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
        AddDllDirectory(runtimeDirectory);
    }

    return Task.CompletedTask;
}

private static string GetNewRuntimeFolder(string runtimeDirectory)
{
    if (Version.TryParse(Path.GetFileName(runtimeDirectory), out var runtimeVersion))
    {
        var basePath = Path.GetDirectoryName(runtimeDirectory)!;

        // Look for the next runtime patch
        for (int i = 1; i < 10; i++)
        {
            var newVersion = new Version(runtimeVersion.Major, runtimeVersion.Minor, runtimeVersion.Build + i);
            var newRuntimePath = Path.Combine(basePath, newVersion.ToString());
            if (Directory.Exists(newRuntimePath))
            {
                return newRuntimePath;
            }
        }
    }

    return null;
}

@vitek-karas
Copy link
Member

@joeloff who might have some ideas about the installer removing previous versions even though they're still in use.

I thought the installer will make sure that no .NET app are running (and if they are force a reboot)?

@Bo-Liu-BL
Copy link

Hi @RobrechtU , the Microsoft support engineer is actively working with us and the product team. So far there is no good solution, and he suggested me to this discussion thread.

@Bo-Liu-BL, it would be great if you could post any new insights should you get any news on the Microsoft ticket. Even some official guidance on how to best deal with the issue would be helpful.

@markoweb2
Copy link

I'm not sure why Microsoft is overcomplicating things and claiming there is no good solution. The solution is really simple for them:

  1. There is no requirement to uninstall the old .NET at the time a new version is installed. It can safely be done during the next reboot. As .NET updates are installed on your regular patch tuesday cycle with the rest of Windows Cumulative updates, which are guaranteed 100% to require a restart, then there is no real security concern of keeping the old .NET around for a few more hours.

  2. And if you insist on persisting with removing the old .NET immediately, even then the solution is just ~10 lines of simple code.
    The code loading .dll files lies somewhere in the System namespace. Find it, add a try-catch block to catch the FileNotFoundException. In the catch block, write an if statement:
    If dll path = C:\Program Files\dotnet\shared\....
    then check if there is a newer subfolder. For example the original path requested folder 6.0.21. If there is a now a folder 6.0.24, then use that instead and load the dll file from there.
    If still unsuccesfull, then throw the FileNotFoundException.

@antmjones
Copy link

We have encountered this issue twice now (in a IIS hosted web app, using in-proc), once updating 7.0.12 -> 7.0.13, and more recently with 8.0.3 -> 8.0.4.

In our case, this resulted in the following exception being logged by IISHttpContextOfT.ReportApplicationError:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.AspNetCore.Diagnostics.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

What appears to have happened is that the web application did restart automatically (due to ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange), but this restart happens before all of the relevant msi packages installed. In particular, the app restarted before aspnetcore-runtime-8.0.4-servicing.24170.14-win-x64.msi was installed, which presumably led to it picking up the old 8.0.3 aspnetcore files (presumably via finding 8.0.3/Microsoft.AspNetCore.App.deps.json at startup).

In the logs, we can see it restarting with mixed versions of .net (note 8.0.4 for hostxfr vs. 8.0.3 for aspnetcorev2_inprocess):

[2024-04-10T04:47:24.520Z, PID: 8760] [aspnetcorev2.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\8.0.4\hostfxr.dll
...
[2024-04-10T04:47:24.674Z, PID: 8760] [aspnetcorev2.dll] Loading request handler:  'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\8.0.3\aspnetcorev2_inprocess.dll'

This wouldn't be an major issue if the app had then managed to restart again after aspnetcore-runtime-8.0.4-servicing.24170.14-win-x64.msi installed, but instead we see event log entries of the form:

Application 'C:\Windows\System32\inetsrv\w3wp.exe' (pid 8760) cannot be restarted - Application SID does not match Conductor SID

This means that the web application ends up with the asp.net dlls being removed from underneath it, which leaves it in a completely broken state until it is forcibly restarted. We could disable automatically installing .net updates but would prefer not to have to do these manually to avoid potentially missing security updates. For us, addressing the failure to restart the IIS worker process would be sufficient to "fix" the issue.

@FunLow
Copy link

FunLow commented Jun 3, 2024

As described in #102804 we are facing a similiar issue related to missing DLL's after enabling and using the automatic update. Are there any news or plans to fix this issue?
As described in the issue we would like to use the method to patch all our .NET based virtual machines. But in the current state this leads only to outages on a lot of systems including productive ones.

@ueli-werner
Copy link

ueli-werner commented Jun 4, 2024

A major, easy-to-fix problem that has not been solved since 2021.

@TiltonJH
Copy link

TiltonJH commented Jul 23, 2024

We are running into the same issues with our clients as well as our servers.

Does anyone know a workaround?

@RobrechtU
Copy link

We are running into the same issues with our clients as well as our servers.

Does anyone know a workaround?

@TiltonJH We are still using the workaround I mentioned in an earlier comment and have not run into the issue in production since releasing it several months ago.

@awakecoding
Copy link

Hi, I work for Devolutions on Remote Desktop Manager, and since switching from .NET Framework to .NET 8, we've been having random crashes in our application every single time a .NET runtime security runtime update was pushed. The reason? Windows Update pulling the rug under our running application, causing the next assembly loading to fail because the previous copy of the .NET runtime has been removed during the background update. .NET 8.0.8 was released yesterday, so today we're already getting error reports from users that launched RDM with .NET 8.0.7 only to see Windows Update automatically update to .NET 8.0.8 in the background.

I tried reproducing the issue manually using the .NET 8.0.7 and 8.0.8 installers, but I noticed that older copies of the .NET runtime remain, so there's no rug pulling happening like with Windows Update. By this I mean that if you install .NET 8.0.7 with the installer, you'll have a directory like 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.7'. Updating to .NET 8.0.8 will create a new directory like 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.8' but it won't delete the previous one, so running applications will not hit problems due to missing assemblies.

I downgraded to .NET 8.0.7 again, then ran Windows Update to let the update happen in the background this time, and I was able to reproduce the issue this way. Unlike running the installer manually, Windows Update removes older copies of the .NET runtime. I understand there is a desire not to leave old files lying around, but this is exactly what happens with the current installer anyway. I would suggest a compromise: why not simply delete the old copy of the .NET runtime on the next reboot? This way, no running apps will ever be affected.

Last but not least, here's an analysis of the problem and related steps, with screenshots:

Launch Remote Desktop Manager with .NET 8.0.7, then check for Windows Update to install .NET 8.0.8:

image

Do something with Remote Desktop Manager until it tries to load an assembly it hasn't already loaded, causing a crash:

image

Looking into the .NET runtime directories, we can see only the 8.0.8 directory remains, the old 8.0.7 directory is gone:

image

In Process Monitor, we can see Windows Update moving the older 8.0.7 files to a temporary rollback location:

image

That last bit is what had me puzzled. You can't delete files that are in use by other applications, so how could the entire directory be gone after Windows Update did its thing? That's because they're moved and possibly just deleted afterwards.

The problem is that a currently running application will keep looking in the old directory that no longer exists to load new assemblies, and it cannot possibly find them when they've been moved like this.

What do you think of my suggestion to have the previous copy of the .NET runtime simply scheduled for deletion on the next reboot? The files need to remain in place for a running .NET application to work properly, at least until the application is restarted. The current situation of "yeah well just restart your application if it crashes after a .NET security update through Windows Update" is not acceptable IMHO.

@logicaloud
Copy link

Do not trust to hope; it has forsaken this issue.

@joeloff
Copy link
Member

joeloff commented Nov 13, 2024

Hi

Starting with .NET 8 and 9 today, you can now set a registry key that will defer the removal of .NET until the next session. We're busy updating the documentation for this. This will help specifically with situations where an FDD application is left running during an update and then crash because the runtime was removed and it tries to load an assembly from a non-existing runtime location.

.NET Version Registry Key Name Type Value
All HKLM\SOFTWARE\Microsoft\.NET RemovePreviousVersion REG_SZ always, never, or nextSession
.NET 9 HKLM\SOFTWARE\Microsoft\.NET\9.0 RemovePreviousVersion REG_SZ always, never, or nextSession
.NET 8 HKLM\SOFTWARE\Microsoft\.NET\8.0 RemovePreviousVersion REG_SZ always, never, or nextSession

@antmjones
Copy link

antmjones commented Nov 13, 2024

Hi @joeloff, could you confirm that those registry key paths are correct? The registry keys for configuring automatic updates are documented as being under "Microsoft\.NET", not "Microsoft.NET" (note the backslash), I would assume that the new entries would be in the same location?

@antmjones
Copy link

In addition, is there any plan of either fixing or providing a workaround with one of the underlying issues when the app is hosted in IIS, i.e. that the app isn't correctly restarted (see my previous comment)? Although the above workaround to prevent the previous version from being immediately removed will avoid the IIS hosted application immediately breaking after an update, it would be preferable if it did just restart correctly so that it will use the latest installed version of .net (as opposed to being stuck on the previous version with potential security vulnerabilities). I can raise a separate issue for this if required?

@KalleOlaviNiemitalo
Copy link

KalleOlaviNiemitalo commented Nov 13, 2024

@antmjones, you're right, the installer actually expects a backslash between Microsoft and .NET: HKLM\SOFTWARE\Microsoft\.NET.

Some related pull requests:

The dotnet/arcade PRs add code to search for the "RemovePreviousVersion" registry values and set the RemoveUpgradeRelatedBundle variable in the installer. The dotnet/sdk PRs add code to include those registry searches in the installer and set the version numbers that are used for the Registry keys. But I don't understand what finally reads the RemoveUpgradeRelatedBundle variable.

@joeloff
Copy link
Member

joeloff commented Nov 13, 2024

@antmjones, you're right, the installer actually expects a backslash between Microsoft and .NET: HKLM\SOFTWARE\Microsoft\.NET.

Some related pull requests:

The dotnet/arcade PRs add code to search for the "RemovePreviousVersion" registry values and set the RemoveUpgradeRelatedBundle variable in the installer. The dotnet/sdk PRs add code to include those registry searches in the installer and set the version numbers that are used for the Registry keys. But I don't understand what finally reads the RemoveUpgradeRelatedBundle variable.

The last changes will be made public soon - the final change was to modify and rebuild the engine (burn) in with as well as the standard bootstrapper application that runs on to (wixstdba). The BA ends up reading the variable, then modifies the installation plan when upgrade related bundles are detected. So either nothing happens, the default action is taken or when deferring, the engine the writes the RunOnce key

@joeloff
Copy link
Member

joeloff commented Nov 13, 2024

Hi @joeloff, could you confirm that those registry key paths are correct? The registry keys for configuring automatic updates are documented as being under "Microsoft.NET", not "Microsoft.NET" (note the backslash), I would assume that the new entries would be in the same location?

Sorry, meant to reply and looks like I hit edit on your comment. Thanks for pointing out the typo. I missed escaping the slash. Table is updated now

@joeloff
Copy link
Member

joeloff commented Nov 14, 2024

@KalleOlaviNiemitalo Here's the primary change we made to the setup engine: dotnet/wix3@8457015

@LukeDearden
Copy link

Hi

Starting with .NET 8 and 9 today, you can now set a registry key that will defer the removal of .NET until the next session. We're busy updating the documentation for this. This will help specifically with situations where an FDD application is left running during an update and then crash because the runtime was removed and it tries to load an assembly from a non-existing runtime location.
.NET Version Registry Key Name Type Value
All HKLM\SOFTWARE\Microsoft.NET RemovePreviousVersion REG_SZ always, never, or nextSession
.NET 9 HKLM\SOFTWARE\Microsoft.NET\9.0 RemovePreviousVersion REG_SZ always, never, or nextSession
.NET 8 HKLM\SOFTWARE\Microsoft.NET\8.0 RemovePreviousVersion REG_SZ always, never, or nextSession

Will there be an equivalent option for dotnet on Linux?

Re dotnet/sdk#42445

@mzros
Copy link

mzros commented Nov 14, 2024

Is there a reason why “always” is used as the default and not nextSession? If nextSession were used for this, no runtime errors would occur after .NET updates.

wix installer

@ueli-werner
Copy link

Using always as default results in broken by default.

@joeloff
Copy link
Member

joeloff commented Nov 14, 2024

@mzros, @ueli-werner The majority of customers expect updates to remove previous versions, especially when automatic updates are involved on Windows. Using nextSession as the defaultd was a breaking change we did not want to spring on people. With nextSession, because it writes to RunOnce, standard users would not be able to trigger the removal since they don't have admin tokens. It would require an admin to deploy manual cleanup scripts using tools like SCCM.

@LukeDearden nothing equivalent on Linux - the majority of issues I've come across was specific to Windows, but I'll take a peek at the Linux one.

@ueli-werner
Copy link

@joeloff There may be reasons for this implementation, but don't tell me that the majority of your customers expect an update to remove the framework of a running application, resulting in a crash without countermeasures. The only reasonable way for an application is to lock all those files. Since we have no control over our customers systems, the registry hack is only applicable in a few cases.

@logicaloud
Copy link

Good to see some updates coming through.

The good thing is that, should an application developer become aware of this issue (s)he can immediately advise the customer to change the registry setting to "nextSession" and avoid further crashes without modifying the application.

An application developer that is aware of the issue may also advise the customer in advance to use the "nextSession" registry setting. The customer may not be known in advance though, and it is at least "awkward" to ask known customers.

The bad thing is that, with a default of "always", application developers will continue to log unexplained issues before they find their way here, and the application needs to survive the initial aura of unreliability.

Overwriting the registry key at the application's installation time and force it to "nextSession" seems to be a bit of an overreach, and it does not protect against registry key changes in the future. One could argue that locking all files during the application's runtime is the same overreach; the difference is that, if the application with locked files is not running or if it is uninstalled, then the previous .NET version can be removed immediately.

Long running applications are unfortunately the ones that have the most impact should they fail, and for those at least, locking files seems to remain the best option to reliably prevent future crashes, independent of the registry setting at the time. That assumes of course that an "always" setting does not override the file locking and kill the application.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests