-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DllNotFoundException after .NET 5 in-place update #60144
Comments
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
@NikolaMilosavljevic any idea where this issue should go? WPF? |
This can stay in runtime as it is something that can be experienced with shared and desktop runtimes. |
OK picked area-Setup. |
@danmoseley Is there an actual time table, when we can expect some feedback on this? We are in the middle of the release of our biggest update in years (migration from 4.7.2 to 5.0) and as the first small wave of customers got the update, this issue appeared and it seems huge. |
❗ @NikolaMilosavljevic @vitek-karas can you help find someone to localize this one? It is blocking @rkieslinger update. |
@joeloff for the installer side of things. I don't think there's much the runtime can do here. If there's a way for the setup to leave the old version in place and only remove it upon next restart - that would probably be the best solution. But I don't know what is the contract with security servicing. |
If I remember correctly, the .NET Runtime update actually did install side by side, but the updates of the .NET Desktop Runtime and ASP.NET Core Runtime did not and so they are the ones that are causing issues. @TRZLGast can you confirm that, just to be sure? |
I don't see much we can do from the setup side either. Upgrades uninstall. There's no way to defer a part of the upgrade until the next restart. Right now we install the new version and remove the old. That would break 99% of the cases where leveraging MU is removing the previously vulnerable pieces from the machine. We introduced upgrades around 3.0 across the board. There are two models being used.
This would be less of an issue for .NET Framework because assemblies would end up in the GAC I think |
So to sum it up, we just have these 2 options?
Doesn't MSI sometimes utilize Another approach which gets to my mind could be an entry in |
Yes that's right. |
How did you acquire these? The runtime installers will always upgrade, but SDK feature bands are SxS, so if you installed SDK 5.0.100 and then 5.0.101, that will upgrade the runtime from 5.0.0 to 5.0.1. However, you could then install SDK 5.0.205. the 2xx and 1xx bands are SxS, so that would allow you to have both the 5.0.1 and 5.0.8 runtimes. If you installed just the runtime, e.g. But if you installed |
Adding @dleeapho since this is mostly about runtime installers. |
+1 for this |
For upgrade dotnet-hosting OPT_NO_X86 and capture with procmon:
Seems killall |
I've also been seeing this randomly on an app (.NET5 Desktop), it's pretty annoying. I was suspecting that it had to do with .NET update, now I have confirmation. Keeping the old .NET runtime seems like the best thing to do. I personnally have many runtimes installed in that folder: So I'm not sure why it has to be different with Microsoft Update. We don't want it to grow indefinitely? |
@rkieslinger I guess another option would be to deploy a self-contained app to your customers. |
We evaluated that before migrating to .NET 5.0+, but decided to roll with a shared framework for couple reasons. We'll stick to that decision and implemented some error management so the user at least gets an info that he has to restart his computer. |
As it happens only on .dll not loaded yet (in our case it often happens on automation DLL for instance) I was also thinking to preload them as startup. |
New runtime versions were just released: And we saw similar expections popping and it's clearly not a coincidence. I was thinking of a new approach by using Something like: public App()
{
var runtimeDirectory = System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory().TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
AppDomain.CurrentDomain.AssemblyResolve += (_, args) =>
{
// Runtime directory is not available anymore, probably deleted by an auto-update, hopefully a new patch should be available
if (!Directory.Exists(runtimeDirectory) && Version.TryParse(Path.GetFileName(runtimeDirectory), out var runtimeVersion))
{
var basePath = Path.GetDirectoryName(runtimeDirectory)!;
// Look for the next runtime patch and load the .dll from there
for (int i = 1; i < 10; i++)
{
var newVersion = new Version(runtimeVersion.Major, runtimeVersion.Minor, runtimeVersion.Build + i);
var newAssemblyPath = Path.Combine(basePath, newVersion.ToString(), new AssemblyName(args.Name).Name + ".dll");
if (File.Exists(newAssemblyPath))
{
return Assembly.LoadFrom(newAssemblyPath);
}
}
}
return null;
};
} NB: I'm not sure how Microsoft Updates manage to remove the old folder, if I try to do it manually, explorer will prevent me from doing so as it's being in use. |
It won't work in the current state:
|
I'm seeing a very similar issue, if not the same. I have encountered this issue with a fully self contained .net core app with WPF elements.
StackTraceString: |
Looks indeed like it! Please note that it's hard to reproduce as you need to uninstall .NET while your app is running. Over than that, it seems pretty consistent. Some users reported that it happened with Visual Studio Installer. I tried a workaround by watching the file changes on the .NET folder and "swap" to the new one but it never kicked in. It looks like that: // Get WPF runtime directory
var runtimeDirectory = Path.GetDirectoryName(typeof(Window).Assembly.Location);
var sharedRuntimePath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles), "dotnet", "shared", "Microsoft.WindowsDesktop.App");
// Watch parent folder for modifications
if (runtimeDirectory.StartsWith(sharedRuntimePath))
{
_fileWatcherService.AddWatcher(sharedRuntimePath, OnFolderChangedAsync);
}
Task OnFolderChangedAsync(object sender, FileSystemEventArgs e)
{
if (!_runtimeDeleted && !Directory.Exists(runtimeDirectory))
{
// Runtime was deleted 😲
_runtimeDeleted = true;
// Track exception so that we're aware when all this gets triggered
_telemetryService.TrackException(new InvalidOperationException(".NET Runtime was deleted"));
_logger.LogWarning($".NET Runtime was deleted ({runtimeDirectory})");
// Ask the user to restart praying that he/she clicks on it
_notificationManager.ShowRestartNotification(".NET Runtime was updated. You need to restart the application.");
// Try resolving the new framework version
runtimeDirectory = GetNewRuntimeFolder(runtimeDirectory);
if (runtimeDirectory == null)
{
_logger.LogError(".NET Runtime couldn't be resolved to a new directory.");
return Task.CompletedTask;
}
_logger.LogWarning($".NET Runtime was resolved to new location: {runtimeDirectory}.");
// Hook up on AssemblyResolve for managed assemblies
AppDomain.CurrentDomain.AssemblyResolve += (_, args) =>
{
var newAssemblyPath = Path.Combine(runtimeDirectory, new AssemblyName(args.Name).Name + ".dll");
if (File.Exists(newAssemblyPath))
{
return Assembly.LoadFrom(newAssemblyPath);
}
return null;
};
// Update default directories for native .dll
SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
AddDllDirectory(runtimeDirectory);
}
return Task.CompletedTask;
}
private static string GetNewRuntimeFolder(string runtimeDirectory)
{
if (Version.TryParse(Path.GetFileName(runtimeDirectory), out var runtimeVersion))
{
var basePath = Path.GetDirectoryName(runtimeDirectory)!;
// Look for the next runtime patch
for (int i = 1; i < 10; i++)
{
var newVersion = new Version(runtimeVersion.Major, runtimeVersion.Minor, runtimeVersion.Build + i);
var newRuntimePath = Path.Combine(basePath, newVersion.ToString());
if (Directory.Exists(newRuntimePath))
{
return newRuntimePath;
}
}
}
return null;
} |
@joeloff who might have some ideas about the installer removing previous versions even though they're still in use. I thought the installer will make sure that no .NET app are running (and if they are force a reboot)? |
Hi @RobrechtU , the Microsoft support engineer is actively working with us and the product team. So far there is no good solution, and he suggested me to this discussion thread.
|
I'm not sure why Microsoft is overcomplicating things and claiming there is no good solution. The solution is really simple for them:
|
We have encountered this issue twice now (in a IIS hosted web app, using in-proc), once updating 7.0.12 -> 7.0.13, and more recently with 8.0.3 -> 8.0.4. In our case, this resulted in the following exception being logged by
What appears to have happened is that the web application did restart automatically (due to In the logs, we can see it restarting with mixed versions of .net (note 8.0.4 for hostxfr vs. 8.0.3 for aspnetcorev2_inprocess):
This wouldn't be an major issue if the app had then managed to restart again after
This means that the web application ends up with the asp.net dlls being removed from underneath it, which leaves it in a completely broken state until it is forcibly restarted. We could disable automatically installing .net updates but would prefer not to have to do these manually to avoid potentially missing security updates. For us, addressing the failure to restart the IIS worker process would be sufficient to "fix" the issue. |
As described in #102804 we are facing a similiar issue related to missing DLL's after enabling and using the automatic update. Are there any news or plans to fix this issue? |
A major, easy-to-fix problem that has not been solved since 2021. |
We are running into the same issues with our clients as well as our servers. Does anyone know a workaround? |
@TiltonJH We are still using the workaround I mentioned in an earlier comment and have not run into the issue in production since releasing it several months ago. |
Hi, I work for Devolutions on Remote Desktop Manager, and since switching from .NET Framework to .NET 8, we've been having random crashes in our application every single time a .NET runtime security runtime update was pushed. The reason? Windows Update pulling the rug under our running application, causing the next assembly loading to fail because the previous copy of the .NET runtime has been removed during the background update. .NET 8.0.8 was released yesterday, so today we're already getting error reports from users that launched RDM with .NET 8.0.7 only to see Windows Update automatically update to .NET 8.0.8 in the background. I tried reproducing the issue manually using the .NET 8.0.7 and 8.0.8 installers, but I noticed that older copies of the .NET runtime remain, so there's no rug pulling happening like with Windows Update. By this I mean that if you install .NET 8.0.7 with the installer, you'll have a directory like 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.7'. Updating to .NET 8.0.8 will create a new directory like 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\8.0.8' but it won't delete the previous one, so running applications will not hit problems due to missing assemblies. I downgraded to .NET 8.0.7 again, then ran Windows Update to let the update happen in the background this time, and I was able to reproduce the issue this way. Unlike running the installer manually, Windows Update removes older copies of the .NET runtime. I understand there is a desire not to leave old files lying around, but this is exactly what happens with the current installer anyway. I would suggest a compromise: why not simply delete the old copy of the .NET runtime on the next reboot? This way, no running apps will ever be affected. Last but not least, here's an analysis of the problem and related steps, with screenshots: Launch Remote Desktop Manager with .NET 8.0.7, then check for Windows Update to install .NET 8.0.8: Do something with Remote Desktop Manager until it tries to load an assembly it hasn't already loaded, causing a crash: Looking into the .NET runtime directories, we can see only the 8.0.8 directory remains, the old 8.0.7 directory is gone: In Process Monitor, we can see Windows Update moving the older 8.0.7 files to a temporary rollback location: That last bit is what had me puzzled. You can't delete files that are in use by other applications, so how could the entire directory be gone after Windows Update did its thing? That's because they're moved and possibly just deleted afterwards. The problem is that a currently running application will keep looking in the old directory that no longer exists to load new assemblies, and it cannot possibly find them when they've been moved like this. What do you think of my suggestion to have the previous copy of the .NET runtime simply scheduled for deletion on the next reboot? The files need to remain in place for a running .NET application to work properly, at least until the application is restarted. The current situation of "yeah well just restart your application if it crashes after a .NET security update through Windows Update" is not acceptable IMHO. |
Do not trust to hope; it has forsaken this issue. |
Hi Starting with .NET 8 and 9 today, you can now set a registry key that will defer the removal of .NET until the next session. We're busy updating the documentation for this. This will help specifically with situations where an FDD application is left running during an update and then crash because the runtime was removed and it tries to load an assembly from a non-existing runtime location.
|
Hi @joeloff, could you confirm that those registry key paths are correct? The registry keys for configuring automatic updates are documented as being under "Microsoft\.NET", not "Microsoft.NET" (note the backslash), I would assume that the new entries would be in the same location? |
In addition, is there any plan of either fixing or providing a workaround with one of the underlying issues when the app is hosted in IIS, i.e. that the app isn't correctly restarted (see my previous comment)? Although the above workaround to prevent the previous version from being immediately removed will avoid the IIS hosted application immediately breaking after an update, it would be preferable if it did just restart correctly so that it will use the latest installed version of .net (as opposed to being stuck on the previous version with potential security vulnerabilities). I can raise a separate issue for this if required? |
@antmjones, you're right, the installer actually expects a backslash between Microsoft and .NET: Some related pull requests:
The dotnet/arcade PRs add code to search for the "RemovePreviousVersion" registry values and set the |
The last changes will be made public soon - the final change was to modify and rebuild the engine (burn) in with as well as the standard bootstrapper application that runs on to (wixstdba). The BA ends up reading the variable, then modifies the installation plan when upgrade related bundles are detected. So either nothing happens, the default action is taken or when deferring, the engine the writes the RunOnce key |
Sorry, meant to reply and looks like I hit edit on your comment. Thanks for pointing out the typo. I missed escaping the slash. Table is updated now |
@KalleOlaviNiemitalo Here's the primary change we made to the setup engine: dotnet/wix3@8457015 |
Will there be an equivalent option for dotnet on Linux? |
Is there a reason why “always” is used as the default and not nextSession? If nextSession were used for this, no runtime errors would occur after .NET updates. |
Using always as default results in broken by default. |
@mzros, @ueli-werner The majority of customers expect updates to remove previous versions, especially when automatic updates are involved on Windows. Using nextSession as the defaultd was a breaking change we did not want to spring on people. With nextSession, because it writes to RunOnce, standard users would not be able to trigger the removal since they don't have admin tokens. It would require an admin to deploy manual cleanup scripts using tools like SCCM. @LukeDearden nothing equivalent on Linux - the majority of issues I've come across was specific to Windows, but I'll take a peek at the Linux one. |
@joeloff There may be reasons for this implementation, but don't tell me that the majority of your customers expect an update to remove the framework of a running application, resulting in a crash without countermeasures. The only reasonable way for an application is to lock all those files. Since we have no control over our customers systems, the registry hack is only applicable in a few cases. |
Good to see some updates coming through. The good thing is that, should an application developer become aware of this issue (s)he can immediately advise the customer to change the registry setting to "nextSession" and avoid further crashes without modifying the application. An application developer that is aware of the issue may also advise the customer in advance to use the "nextSession" registry setting. The customer may not be known in advance though, and it is at least "awkward" to ask known customers. The bad thing is that, with a default of "always", application developers will continue to log unexplained issues before they find their way here, and the application needs to survive the initial aura of unreliability. Overwriting the registry key at the application's installation time and force it to "nextSession" seems to be a bit of an overreach, and it does not protect against registry key changes in the future. One could argue that locking all files during the application's runtime is the same overreach; the difference is that, if the application with locked files is not running or if it is uninstalled, then the previous .NET version can be removed immediately. Long running applications are unfortunately the ones that have the most impact should they fail, and for those at least, locking files seems to remain the best option to reliably prevent future crashes, independent of the registry setting at the time. That assumes of course that an "always" setting does not override the file locking and kill the application. |
Description
If Microsoft Update (MU) installs an in-place update of .NET 5 framework (no major/minor feature update) while framework dependent applications are running, then these applications may crash after a time because of missing dlls (FileNotFoundException/DllNotFoundException).
The in-place update installs the new version of the .NET framework and then it tries to delete the old version. If dlls of the old version have been loaded, they can't be deleted and so they were moved to a temporary hidden directory that will be deleted after restart. Moved dlls stay loaded in running applications, but can't be loaded by name/path anymore. And that's exactly the problem.
If one of these applications need to load an additional framework dll, the dotnet loader looks for this dll in the framework directory of the old version. This directory doesn't exist and so the application crashes because of DllNotFound or FileNotFound exception
Steps to reproduce
Application: WpfApp1.exe
CoreCLR Version: 5.0.721.25508
.NET Version: 5.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException: Could not load file or assembly 'System.Printing, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.
With Procexp you can see the movement of loaded dlls.
Before .NET framework update
after .NET framework update
With Procmon you can see where the loader is searching for framework dlls after .NET framework update:
But this directory doesn't exist anymore.
You can reproduce this problem even easier. After an update of the .NET framework you only need to close the WPF application (no button click or custom code necessary). The application will crash again, but now you doesn't recognize it directly. You have to look into event log:
Application: WpfApp1.exe
CoreCLR Version: 5.0.721.25508
.NET Version: 5.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.DllNotFoundException: Unable to load DLL 'vcruntime140_cor3.dll' or one of its dependencies: Das angegebene Modul wurde nicht gefunden. (0x8007007E)
at __std_type_info_destroy_list(__type_info_node* )
at __scrt_uninitialize_type_info()
at _app_exit_callback()
at .LanguageSupport.DomainUnload(Object A_0, EventArgs A_1)
at .ModuleUninitializer.SingletonDomainUnload(Object source, EventArgs arguments)
at System.AppContext.OnProcessExit()
Because Window Update is running behind the scenes, and the described problems occur much later, it’s not easy to comprehend why the applications are crashing. .NET framework will be updated monthly and so there is a high risk that these types of problems appear.
The text was updated successfully, but these errors were encountered: