Support long file names on Windows #14062
Comments
|
A bit of classic reading from the BCL team for anyone digging into this issue: Any creative workarounds for the path normalization issue? |
This will never happen, and should never happen. Please do not add implicit long path support to .NET, it will be a nightmare for users, who now have files that they have created in one app are inexplicably unopenable in other apps. MAX_PATH being a #define in Win32 means that it is hard-coded into tens of thousands of applications, in a way that is almost impossible to change after-the-fact (i.e. via AppCompat shims, etc). You can't even set this in new versions of the SDK because a bunch of apps have made wire / IPC protocols which would include MAX_PATH in the definition, and now they disagree and you've broken things. Using extended syntax also bypasses all of the path correction that Win32 does and means that path semantics are now different between apps, depending on their .NET version. Attempting to "fix" MAX_PATH is simply not tenable in any sane way, especially from a non-OS component like the CLR. |
|
The reality is that you already have tons if files that are unopenable by .Net applications because the framework doesn't let you bypass MAX_PATH. Or you have to resort to rewriting System.IO classes using P/Invoke like in the case of AlphaFS. This should be possible to enable long path support from within .Net. Should it be implicit? I could be convinced either way. However, it is possible in Windows and should at least be possible in .Net. |
|
@paulcbetts I believe that this change is really needed because backwards-compatibility with some ancient software is actually not a goal. Even worse, if this kind of backwards compatibility is enforced in such ways. And that posts by BCL team state that they understood that enforcing MAX_PATH is not really long-term option even back in 2007. |
The Windows 8.1 SDK defines MAX_PATH to be 260 characters. That's brand new software, written today, that has all of the issues that I described. The situation between 15 years ago when Win2k came out, and software written 10 minutes ago, is 100% identical - trying to fix this won't break ancient software, it'll break software written today too |
|
Yes, Windows 8.1 SDK has MAX_PATH defined to 260. And this doesn't mean that we shouldn't support something new only because old applications will not be able to open resulting files. Using this logic we might as well revert back to DOS and FAT16, because some old applications don't work well with Windows and non-8.3 names. |
|
Whatever you choose, we're right here: |
|
Yes Windows defines MAX_PATH at 260 but it at least gives you a means to go beyond that if you need to and has for a number of years. Today, core .Net gives you no options but to bypass System.IO almost entirely and drop down to P/Invoke. It is pointless to debate whether or not you should be able to go beyond MAX_PATH or whether that will break backwards compatibility because Windows already decided that you can. The question then becomes "how can you access those files from within .Net?" Today you can't from the BCL. Your only option is to use something like AlphaFS, which is a great library but it solves a problem that frankly shouldn't exist in .Net. |
|
To make sure I'm super clear- I'm not suggesting that .NET return ?\ paths. Just that it convert internally when talking to Win32 APIs that require it and strip the prefix off again. I struggled with all of this years ago when I was writing the project system for Expression Blend. It was super common to get relative paths that were longer than 260 characters when combined with their base source location (notably where the project file is). While you could set the working directory, it isn't wise as any other thread can also do so (consistent bug for an inconsistent one isn't the best of trades). I discovered that Win32's GetFullPathName could care less about the path length (at least on XP and later) and made my own call to attempt to get the path name below 260 before letting the System.IO.Path.GetFullPath method get it. (System.IO does a little extra normalizing, primarily expanding all path segments that are expandable from 8.3, even if the full path doesn't exist.) The ability of GetFullPathName to take long strings is a part of the answer to the normalization issue mentioned above. But even if this didn't work, the normalization that gets skipped by ?\ isn't that terribly hard to replicate. I'm happy to contribute towards snuffing out the MAX_PATH and other Path.IO related problems. |
|
System.IO is in mscorlib. Shouldn't this issue be tracked at CoreCLR instead? |
|
@poizan42 see dev guide for coreclr - it states that API review should happen in corefx, and this item definitely requires review |
|
@Alexx999 well I think it says that cases which requires changes to mscorlib will be addressed on a case-by-case basis. |
|
The relevant IO stuff is no longer in mscorlib for .NET Core, it's in System.IO.FileSystem and that's already part of this repository: https://github.com/dotnet/corefx/tree/master/src/System.IO.FileSystem/src/System/IO |
|
What exactly is the deal with mscorlib System.IO vs. System.IO.FileSystem? Looking at the implementation in FileSystem is what caused me to kick off this thread here. System.IO.FileSystem seems mostly isolated from mscorlib's System.IO and implements most of the functionality (GetFullPath being one of the few calls I see going to System.IO.Path). |
|
MSCorlib contains the old desktop & phone implementations. We are trying to move as much as possible out of mscorlib that doesn't need to be there. Getting things out of mscorlib lets us ship them independently and enables more pay-for-play eventually when we can slim down mscorlib. The long path work will need to touch the Path class primarily, which is still in mscorlib, but we do have plans to pull it out. I'd prefer that we wait for that to happen so that we can do this potentially destabilizing work in isolation without risking the phone & desktop platforms which ship the mscorlib implementation inbox. This is all good discussion. I agree that we need to support long paths. It will be done and must be done in a way that doesn't break existing apps. We'll likely have an implementation that lets you put an app in "max path mode" that would put up guard rails needed if you are working with legacy components that aren't long path aware. We'd need to identify the trigger for such a mitigation (opt-in-api vs opt-out-api vs quirk) and the right scope (app, context, or instance). I'd be curious to hear folks opinion on this. |
|
I'm not sure how helpful the Framework can be for interacting with components that don't handle >MAX_PATH. Ultimately there is nothing preventing you from sending a really long path to some random component. While I can imagine that IO.FileSystem throwing PathToLong might help in some cases- that seems more likely to be accidental than intentional. If callers do robust error handling on System.IO, why wouldn't they do it on other, presumably more sensitive, components? I guess one argument would be around mitigating buffer overruns in unmanaged components. If the framework throws you're probably less likely to get to the unmanaged component. That is, of course, presuming code usually hits some System.IO API beforehand. One thought is that having iterators (GetFiles) never return long paths might be a reasonable compat option to have? Outside of that Path.Combine and Path.GetFullPath are the most likely choke points. |
|
@JeremyKuhne I don't think that having restrictions on select APIs will really help - I don't see real difference between long path from iterator vs long path from some user input. It may go to some native code or be written to some location where it will cause problems anyway. |
|
@Alexx999 That is sort of my main point. I don't really think restrictions would be that useful as there are no guarantees people will call them on all paths before passing them on. And, to be clear, I would not recommend any of these APIs do anything but support long filenames by default. Just brainstorming possible opt-in behaviors. Along the lines of mitigation- Kim pointed out that the Windows shell collapses paths to short equivalents to get them to fit under 260. While that is interesting, that behavior breaks the desire for paths to be normalized. In addition, it seems that this feature can be disabled and isn't on by default for all drives. (Anyone know of any links that describes the expected behavior? It seems like only the system drive gets this turned on in Win8- or perhaps there is a drive size gate?) I think it would be useful to provide an API that tries to create these compacted paths for legacy component compat. In addition, potentially provide APIs that help you create hard links (with shorter paths)? |
|
I think the compatibility concerns may be a bit exaggerated. It's already possible to create files with long paths. Granted, the mechanism that can achieve this isn't common but I done this once or twice by accident and end up with files that cannot be opened and that's simply illogical. var longDirPath = @"D:\" + new string('x', 200);
var shortDirPath = @"D:\" + new string('x', 40);
var fileName = "filesdjhaskjdhkashdkjshdkjahsdkahskdhakjsdhkd.txt";
Directory.CreateDirectory(shortDirPath);
File.WriteAllText(Path.Combine(shortDirPath, fileName), "");
Directory.CreateDirectory(longDirPath);
// create a file with a long path...
Directory.Move(shortDirPath, Path.Combine(longDirPath, Path.GetFileName(shortDirPath)));
// ...enumeration works fine...
foreach (var filePath in Directory.EnumerateFiles(longDirPath, "*", SearchOption.AllDirectories)) {
// .. but the file cannot be read because FileStream can't open files with long paths
Console.WriteLine(File.ReadAllText(filePath));
} |
|
it's worth mentioning that the need to deal with long paths is becoming more common. for example, consider visual studio kproj (asp.net 5) projects: they have built-in npm package installation. npm uses recursive paths for dependencies, such that installing even a single package can tip you over 260 - grunt-browserify, last I checked, included long enough paths that not even storing your solution at c:\ would save you. and that's a common package. the implications of this right now are that your build tooling cannot be written in .net if using npm packages, and the git provider for VS doesn't correctly handle the directory structure. it's still a niche use case, but it's growing, and .net will have to provide a way to do this sooner or later. |
|
+1 to supporting long file paths. Lots of issues dealing with node_modules folders that go well beyond 260 chars. |
|
Has any work been done on this yet? I'm not able to open my project in VS 2015 RC:
Log: Can this be made a higher priority? |
|
I'm starting on implementing this support. My initial plan of attack is roughly the following: 1.Allow extended syntax (\?) for common file operations (file info/navigation first, then stream handling) [CoreFX] I'll be opening a variety of subtasks on this over the next day or so. |
|
npm/npm#3697 |
|
👍 thank (deity), finally! |
|
What will this mean for existing .net applications? Will they just start working correctly with long paths once this is implemented? Of particular interest: What about visual studio and msbuild? |
|
+1 to supporting long file paths. I want just add very simple realistic scenario, which shows how files with long path names could be created. Let us you have the following structure of directories: and both So one have to support paths, which are longer as |
|
@mlidbom For existing .NET Desktop apps the hope is to make it "just work" for most apps. First priority, however, is getting implementation in place for .NET Core. We'll then look at exactly how and when we pull support back to the desktop. For VS and MSBuild there may (will probably) be some work that has to be done for existing unmanaged code and P/Invokes. |
|
Extended syntax should now work throughout CoreFx (\?). 🎉 (as of dotnet/corefx#2700) |
|
👍 |
|
that's great news! |
|
Now you can use long paths without extended syntax! (as of dotnet/corefx#3001 & dotnet/corefx#3010) |
|
🎉 |
|
🎉 🏆 |
|
So, when these improvements will be shipped with Windows 10, to bring long path support for existing applications, like Powershell? |
These improvements will be shipped with the next version of the .NET Framework. This has little, to nothing to do with Windows 10. Long path names have been supported on Windows since, at least, Vista -- maybe XP. |
|
@whoisj: Long before XP... Here is an excerpt from the NT4.0 SDK:
The documentation for CreateFile in the NT3.1 SDK does not mention any size restrictions at all, though that's probably an oversight. Nt/ZwCreateFile isn't documented in the NT3.1 WDK, so not hint there about whether the underlying system supported long paths. |
|
Will this be included in .Net 4.6.1? |
|
@smbecker No it will not be in .NET 4.6.1 however we do plan to get it added to a future release of full .NET framework, not sure which one yet. |
|
👍 |
|
https://github.com/dotnet/coreclr/issues/1776 tracks the work for the runtime. CoreFx is fully implemented. |
@whoisj what people generally mean by "support for Windows 10" is that the Explorer and Powershell/Command-Prompt be able to open/delete these files or folders. |
|
@kumarharsh, while I appreciate what you're saying. if people are asking about the Windows Explorer Shell, then they should be looking for avenues to query the owners of the Windows Explorer Shell. The CRL/NetFX developers have little information and influence in that arena. As an aside, if CoreFX supports long file names then it will not be long before PowerShell does as well. |
|
@whoisj I am care about the time: Since CoreFX now supports long path, when will the change published to end users, to make at least PowerShell work? The key question is the “when”. |
|
As I understand it, the process for that is so complicated, that you're First of all, because right now we have this awkward dual-mode .Net where So first they have to actually decide (how) to ship the new CoreFX code in Then they have to decide if it's a minor point release upgrade (like 4.5.1 Then they have to ship that version of .Net in Windows ... but in the Then, if they decided it was a major release of .Net, you have to get a new Of course, in the meantime, on Nano Server they're shipping a Core Joel "Jaykul" Bennett On Wed, Mar 23, 2016 at 4:37 AM, Belleve Invis notifications@github.com
|
|
So the key question turns in “Why so many skilled-guys work so hard wasted-efforts for... They?” Do they hope a sweet? |
|
Hello, I had the same problem. I didn´t know what to do so I searched on the internet for some solutions. And I read about [b]Long Path Tool[/b], which is a great tool in these type of cases. :o It worked really well. Hope it works for you too :p |
|
@whoisj, @be5invis, @Jaykul The anniversary update (RS1) of Windows 10 has PowerShell support for long paths. 4.6.2 is part of that release and PowerShell turns on the .NET functionality. The only caveat is that Windows long path policy needs enabled as (for now) it is off by default. I go over these desktop details in one of my blog posts. Anything that relies on versions of CoreFX will support long paths without any policy changes or other updates to the consuming code as we do the work to make the paths work with ( |
|
@JeremyKuhne interesting that you were looking at this issue, I was just looking too. Have you ever look at this nice write up on long-path support and the canonicalization nightmare? https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html |
|
@whoisj No, I hadn't seen that write up- it is quite good. Thanks for sharing the link. FYI, I did a dump of my own around that time as well: Path Format Overview I've made the various .NET APIs much more friendly to the range of path formats. Mostly by not trying to second-guess what a "bad" path was. :) |
|
@JeremyKuhne I shared your with some folks looking to improve Msys, they seem to think it was rather valuable - so thanks for that! |
ghost
locked as resolved and limited conversation to collaborators
and others
Now would seem to be the time to finally tackle the MAX_PATH nightmare. With the Win32FileSystem and other abstractions we can convert incoming paths to their long path equivalent if needed before handing off to the actual Win32 APIs. It appears that all of the APIs currently in use support the extended syntax (?), with the exception of Get/SetDirectory.
The text was updated successfully, but these errors were encountered: