Node's nested node_modules approach is basically incompatible with Windows

[jonrimmer]

Node needs an alternative approach to endless, recursively nested node_modules folders on Windows. Most Windows tools, utilities and shells cannot handle file and folder paths longer than 260 characters at most. This limit is easily exceeded, and once it is, install scripts start breaking and node_modules folders can no longer be deleted using conventional methods.

This will only get worse as Node packages get more complicated, with bigger and deeper dependency hierarchies. There should be a way to install and run packages that does not use file system recursion to represent the dependency hierarchy.

[Mithgol]

Suggestion:   to mitigate the above mentioned problem, make Node.js version 1.0 (and the corresponding npm) use n_m (3 characters) instead of node_modules (12 characters) by default. This change is expected to double the possible hierarchy depth for modules with names containing nine letters or less.

If a backwards compatibility is necessary, node_modules could still be supported (though not as default). For example,

• require('modulename') could look in node_modules if the module is not found in n_m,
• npm update modulename could kill node_modules/modulename (if it exists) and install n_m/modulename and dependencies.

[OrangeDog]

That doesn't solve the problem, just gives you slightly more breathing room.

The only real solution is going to have all modules at a single level, requiring each other, instead of all having private copies of their dependencies. This would either have to be versioned, and/or module maintainers need to be more careful about breaking changes.
Most package managers work this way, and seem to be getting along nicely.

[indutny]

As an idea: node_modules/module@version

[Mithgol]

The module@version solution eliminates the problem much better than n_m, but not without introducing a bunch of other problems.

Example: if somemodulename@1.2.3 and somemodulename@4.5.6 are both present, what should npm update somemodulename do?

In the same example, what should npm uninstall somemodulename do?

[randunel]

@Mithgol it should probably install somemodule@la.te.st, alongside the others. But npm purge or npm clean should probably be added.

[OrangeDog]

@Mithgol if the latest version is 4.5.6 do nothing, otherwise add the latest version. If you can be clever and determine that a previous version is no longer needed then remove it, otherwise don't worry about it.

@randunel somemodule@latest would be far more sensible if there's an actual need for it, though for backwards-compatibility, I'd guess somemodule would symlink to the latest anyway.

[jdalton]

👍, devs are running into this with individual lodash packages too.
See lodash/lodash#501, gulpjs/gulp-util#23, & twitter/theporchrat.

There looks to be the start of addressing the issue here, but it may need to be used in other methods.
Related to #6891.

[jzaefferer]

Some docs for the code @jdalton linked to: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

[Mithgol]

I guess Node's developers would have to rewrite both require() and npm to use \\?\… paths. If it's possible, 256-character limitation becomes void — even without resorting to n_m or @version.

(Maybe not exactly require() and npm, but rather the entire underlying fs module, also process.cwd(), etc.)

[jonrimmer]

Even if all of Node and NPM are rewritten to use long paths, so long as NPM supports scripts there is a danger of them randomly blowing up when run on Windows due to finding themselves inside a very deep hierarchy then trying to run utilities that don't support long paths.

[Mithgol]

@jonrimmer Many third-party scripts for NPM (such as Grunt, or Mocha, or node-pre-gyp) are Node.js scripts and thus they are to become automagically fixed when Node.js core modules (such as fs and path) and path-related methods (such as process.cwd()) all start using \\?\… unilaterally.

The other utilities would indeed break and thus the community would have to replace them or demand upgrades. (A similar ruckus once happened when Node.js started supporting Windows and many UN*X-only tools and CLI scripts experienced a lot of incompatibility-related problems. There was much suffering; however, the community eventually fixed the problems or developed some workarounds, sometimes as simple as using path.join instead of a former + '/' +. Or using os.EOL instead of a former \n.)

[langri-sha]

Hello everyone! I was unpleasantly surprised when I found out our module was hitting an unexpected wall in Windows host environments. If anyone needs a quick remedy:

• You can delete the Windows paths by slicing out a branch of the Node dependency tree and moving it to a shorter path in the host environment.
• If you are developing on a Windows host, but using a Unix environment in production, you can use Vagrant to create a virtual machine, and run you application in the guest environment.
  If you would like to keep your sources and Node modules synced with the Windows host, you can install lodash in the guest environment and use npm link to resolve it as a module dependency for your application.

[domenic]

FYI Node (and thus npm) always uses UNC paths internally, before actually calling into the filesystem. It is only third-party tools that have a problem. (Unfortunately one of those third-party tools is Windows Explorer, but, that's Microsoft's bug...)

[jdalton]

FYI Node (and thus npm) always uses UNC paths internally

I do see the _makeLong() helper used a lot in fs though is there a chance something was missed?
Is it handled in require?

[jonrimmer]

@domenic Microsoft have said many times that MAX_PATH limitations are not considered a bug, just a feature of Windows. The file system supports long paths, the OS does not, there is a difference.

If Node's intention is to support NTFS, then long paths are fine. If the intention is to support Windows, then there needs to be an alternative to long paths.

[domenic]

@jonrimmer The OS does support long paths, or at least all API methods that libuv uses support long paths. So the distinction you are trying to make does not apply.

@jdalton require delegates to fs.

[jdalton]

@domenic Cool.

I'm experimenting with this on my Windows 8.1 machine (I haven't hit this issue on my own yet):

• I created several nested long name directories and navigated via the command line as far as I could before receiving The filename or extension is too long. errors.
• I then did npm install lodash.template. While I couldn't create further folders in Explorer or navigate via the command line the npm install succeeded and the folders were created.
• Once they were created by npm I could navigate to them with Explorer
  (still not with the command line though).
• In Node, from the directory I performed npm install in, I did require('lodash.template') and it loaded the module without problems.
• I even did require('lodash.template/node_modules/lodash.defaults/node_modules/lodash._objecttypes') and it loaded the module w/o issue.
• I noticed in Explorer when I try to copy the path it gives me a truncated form:
  C:\Users\jdalton\Desktop\A12345~1\A12345~1\A12345~1\A12345~1\NODE_M~1\LODASH~1.TEM\NODE_M~1\LODASH~1.DEF\node_modules\lodash._objecttypes.
• Also if I cd <that truncated path> I can navigate to the path in the command line as well. So now I'm in the directory in the command line and in Explorer (I can also create more folders/files) and the module is loading in Node.

[jonrimmer]

@domenic Microsoft: "In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters." [1]

Some (but not all) Windows APIs support, as an alternative, unicode paths up to 32,767 chars, but this is an alternative - not the default. It is not the option used in many Windows utilities including core parts of the OS. Claiming therefore, that Windows supports long paths, is like claiming old NT was Unix compatible because there is a POSIX subsystem, or claiming Linux is compatible with the win32 APIs because you can install Wine. It is a entirely disingenuous attempt to misrepresent the actual reality. Windows as an OS does not support long paths. That it is possible to write Windows compatible software that supports long paths does not change that fact.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

[domenic]

I don't really care about whatever word games you want to play around the word "support." Sure, if you wish, all I'm saying is that Node and npm are Windows compatible software that support long paths.

[jonrimmer]

You are the one playing games - calling core parts of Windows like Explorer "3rd party tools", and suggesting that not supporting long paths is a bug. Microsoft have made it clear repeatedly that non-support for long paths is not a bug, and not something that will change.

A package manager creating paths that do not work with the majority of the software written for an OS, then claiming compatibility with the OS, is playing games, at your users expense.

[edef1c]

I guess we'll have to stick with just being compatible with node and npm for Windows then.
To argue that the node module system should change due to deficiencies in parts of Windows is a dead end.

[matt-bernhardt]

Well, I guess that writes off my ability to use node (or at least some node projects) under a Windows environment, then? I've been attempting to merge a repository elsewhere on Github that uses node, and the merge command fails due to the length of the pathnames.

Maybe there's a workaround I can use (seriously, I'm open to suggestions) - or maybe you can argue that the problem lies with how the Git command line deals with directory paths - but the end result is the same: I can't work with this node project under Windows.

[OrangeDog]

Nested dependenices shouldn't be in git repositories. Each dependency has its own repository, and npm assembles them.
You could always open a ticket with git to add support for long paths, just like node does.

[dougwilson]

Or you can check out the git repo directly under C: ?

[kleinfreund]

Finished the thread. Is enabling longpaths with git config core.longpaths true the workaround to use? I'm hitting this while installing node packages for Gulp.js.

[innoscience]

Was going nuts with this problem, glad to see I'm not alone. Wanted to add the note that using a VM isn't a solution if your project is in the shared path between your system and the VM (which is generally the case). The same errors occur as the files are not allowed to be written when installing the modules to your project when they hit the limit.

[edef1c]

@innoscience We've seen npm breaking in varied, complicated ways with VM shared paths across different OSes. It's not really a supported thing (and very hard to support unfortunately, as much as I'd love to see that just magically work)

[LoveAndHappiness]

Just wanted to say, how I solved it. I changed to Linux (Zorin Distribution). It costed me nearly two days to set up everything properly and I have a bunch of programs which aren't running, like the Adobe Collection, but at least gulp and yeoman are working, which is the reason for me to use node and npm. Btw, should't this threas been closed?

[phillip-haydon]

The problem hasn't been solved. Just bad work around's. Core issue is unsolved because of reluctance to chance.

[bmeck]

@phillip-haydon is there something that requires the change to the loader over making standard build tools / installation applications work?

[efdee]

@bmeck I'm not sure how you are planning to fix tools that are not open source. Eg: cmd.exe, MSBuild, Explorer.

[bmeck]

@efdee only a concern with install / build phase of modules. Currently working on moving to a TMPDIR approach for building to fix most of those concerns (npm/npm#5523). Already have a PR for auto-enabling core.longpaths when npm shells out to git. Only issue left is the auto "dedup" (npm/npm#4012 ).

[efdee]

@bmeck It's not just an installation/build problem. We ran into errors due to long file names related to the nested node_modules structure when calling Grunt from inside an MSBuild buildfile. If you want I can look up what the exact scenario was.

[StoneCypher]

The amount of time people are spending trying to dust a design flaw under the rug as the operating system's fault is amazing.

It's the world's most common operating system. It doesn't matter whose fault it is. This is a critical and intolerable limitation in Node.

I can't check my Node projects into Git because the Windows Git client uses the default filesystem stuff.

This is because Node took an approach that is antagonistic to the world's most common operating system. This approach is not necessary or efficient.

Refusing to fix it by claiming the thing that was around 30 years earlier is just cheerleading.

Good engineers make tools that are usable, and do not defend unnecessarily broken design strategies as other peoples' fault.

[StoneCypher]

@isaacs No. This is not Microsoft's fault. This is Node's. These design choices were in place before Node existed.

Playing games about wishing Microsoft would support things is beside the point. The Microsoft tools aren't the problem. We can't use you with Git. We can't use you with Bitbucket. We can't use you with compilers, with editors, with network tools.

We can't use you with anything because you broke the rules. Then you said "but there's this other secondary api, and we expect every tool that was ever written to change to conform to us!"

95% of the world's computers, dude. You're being a neckbeard and you're dramatically impeding Node's ability to grow so that you don't have to do a simple algorithm repair and admit you made a mistake.

This is not a legitimate choice.

[matejkramny]

"This is a critical and intolerable limitation in Node.”

No, its a limitation of Windows.

"It's the world's most common operating system”

Of course, and the one developers use the least. In addition, 99% of servers are linux based.

Matej Kramny

On Thursday, 3 July 2014 at 03:50, John Haugeland wrote:

The amount of time people are spending trying to dust a design flaw under the rug as the operating system's fault is amazing.
It's the world's most common operating system. It doesn't matter whose fault it is. This is a critical and intolerable limitation in Node.
I can't check my Node projects into Git because the Windows Git client uses the default filesystem stuff.
This is because Node took an approach that is antagonistic to the world's most common operating system. This approach is not necessary or efficient.
Refusing to fix it by claiming the thing that was around 30 years earlier is just cheerleading.
Good engineers make tools that are usable, and do not defend unnecessarily broken design strategies as other peoples' fault.

—
Reply to this email directly or view it on GitHub (#6960 (comment)).

[phillip-haydon]

Of course, and the one developers use the least. In addition, 99% of servers are linux based.

What absolute rubbish.

[othiym23]

@StoneCypher I understand your frustration, but I do believe this is an issue best handled by npm, simply because it's what's responsible for creating the nested folder structure in the first place. @isaacs and I have spent some time going over the various approaches described here and elsewhere, and I think we're in broad agreement that this problem can be resolved without making changes to the Node module loader.

That said, @isaacs, can you lock this issue now? The issue's been hashed over to death, and the arguing is no longer constructive or useful.

[StoneCypher]

@othiym23 That was my initial reaction too. When I went to them, they were a lot more reasonable than the node people. They recognized that this was a fatal problem. However, their claim is that they have no choice but to use the node deployment system, which was frozen two years after this was first reported.

I tried asking #node.js about it, but I got a bunch of stuff like Matej, acting like if a 45 year old standard and everything written to it doesn't magically mutate to support this new package manager that can't cope with basic filesystem rules, it's everyone else's fault.

It is, actually, constructive and useful, and you should not lock it. The fundamental problem here is cultural: too many node devs are pretending that this isn't hurting people.

Please do not use censorship to suit your goal of pretending that Windows is a first class citizen. Either fix the package manager to work on the filesystem, or stop making the false claim that people are investing lost effort on the basis of.

If the ticket gets locked, people will just open new ones, because this is a very serious problem. Playing locking games leads to more noise, not less. Just unfollow.

[StoneCypher]

Please do not close this ticket. The bug is unnecessary, serious, and fixable.

[phillip-haydon]

Beginning to lose faith in Node, is this is the state of the project then I don't know how anyone can depend on it in production.

[khaledh]

+1 for re-opening the ticket. This issue is not solved yet.

[StoneCypher]

The NPM people state clearly that they have no choice but to follow the pattern that the Node people just said was NPM's fault.

This is not adequate behavior from a tool vendor. Tool vendors must keep high standards.

[StoneCypher]

"To avoid this issue in the next release Lo-Dash core is inlining ~60 functions into various modules which greatly reduces the dep graphs (many to 0 deps) at the cost of duplication. It's a bummer of a band-aid but it's the state of things at the moment."

This is so broken that other libraries are starting to restructure themselves to cope.

[StoneCypher]

@jdalton Is it possible to get the lo-dash that copes with this early?

[othiym23]

@StoneCypher I should have made clear in my earlier response that I work for (and on) npm and am the developer most likely to make whatever changes that the npm team / community deem best to better support Windows users. I want to see Windows users happy, because I think we make a good product and I want everyone to be able to take full advantage of it.

That said, this issue is already closed. It will not be reopened, nor should it be. I think that at this point all of the productive discussion has been exhausted, and instead of new ideas being brought to the table, the discussion is now circular and is doing nothing but frustrating people and causing conflict. Simply insisting that the issue is real, or saying that Windows is simply broken, is not a fruitful debate, it's just cheerleading.

If the Node core team thought that this was an issue that required their involvement, they would have weighed in at some point over the six months that this ticket has been around. Aside from a suggestion by @indutny and @trevnorris wondering why the discussion is happening here and not on npm's issue tracker, they haven't.

Without speaking on anyone's behalf, I'd suggest that's for a very simple reason: Node's module loader is probably the most frozen piece of its API surface, and it's the primary abstraction that connects the 80k+ modules that form the Node module ecosystem. npm has a certain amount of freedom to restructure how it installs modules in projects, as long as it follows the Node module system's rules; if it has that freedom but Node doesn't, why shouldn't it be the piece to change?

It is most definitely not censorship to close out a topic of discussion after the conversation has run out of gas. An issues thread isn't a democratic debate, nor is it a straw poll; it's a place to identify technical flaws and bring them to resolution. As I said earlier, I understand your frustration, and I agree that there is work to be done here. I don't think that either @isaacs or I have done anything that would suggest that we don't take this problem seriously. Beyond that, I don't see any suggestions or proposals in your comments. As you say, you're free to open new issues to keep hashing this out, but at this point, I consider this particular issue an attractive nuisance.

[innoscience]

Pragmatically speaking, if there is a refusal or inability to fix this issue, then this needs to be disclosed plainly and openly to all people planning on using Node so everyone is clear on cross-platform compatibility. Something to the effect that the windows platform is not fully supported due to file path limitations and Node / NPM may fail under certain conditions.

[StoneCypher]

@othiym23 If you choose not to repair or provide a successor to the module system, then please stop making claims of first class windows support.

Javascript libraries are now modifying their own structure to avoid the support requests that come from this design decision of yours.

Nothing's stopping you from implementing a node_mod2 system alongside the existing one to allow new work to transition to a safer model.

Please don't claim support that you don't actually offer. That does economic damage to others who believed you. Thanks.

[StoneCypher]

@othiym23 I'm not entirely sure why you don't just provide a successor and let modules transition. That would be easy and straightforward, would provide no backwards breakage, would allow you to fix the problem that you suggest that you take seriously, would require no effort on your developers' parts other than in the future to put a 2 on some API call, and would also give you the opportunity to apply whatever other lessons you have learned in the meantime.

The reason I didn't offer any solutions is I discovered this ticket where you claimed the problem was Windows' fault, and after you had already made your decision.

I was pleading with you not to turn a blind eye this way.

I'm relatively new to node. I started two days ago, because someone did some work for me in it. This is my first experience with your platform. When I googled it I found dozens of bizarre stackoverflow and reddit posts of the form "help! node put down an illegal directory and I can't even delete it!"

And your response? "Windows is buggy because it behaves the same way it has since three years before JavaScript existed. If Windows doesn't change for us, thereby undermining every other piece of Windows software, then they are in the wrong!"

No, man, they really aren't.

There are tons of answers of what you can do. You're just not going "oh my god, red alarm, everyone is furious and it doesn't work!"

The reason people are speaking to you in the tone that they are is that you give the very strong impression of saying you take things seriously, but in the same thread, you say things like "This is the fault of a company that made this decision before the language we're working in was even created, and we actually genuinely expect them to break everyone else to satisfy us!"

Fundamentally, a boneheaded decision was made at the outset. You won't admit that. As a result, you can't actually get to the point of saying "well, now we need to do something about this."

If you actually wanted to do something about this, it would not be very hard to solve.

But you want to make sure that you don't feel like anyone made any mistakes, and if that means everything was right, then you end up ignoring it for six months then closing it unfixed, while telling yourself how seriously you take it.

We don't care how seriously you take it, if it ends up unfixed at the end of the day, and you end up telling us "go away, we don't want to hear about this anymore."

The problem doesn't end just because you blame someone else and close the ticket, buddy.

You see other people within an hour of calling for silence saying "I don't see how I can take these guys seriously in a production setting."

That's an expensive message to send. Are you really sure you're treating the community in a way that's going to result in them feeling like you're trying to make sure their problems get handled?

I would much rather have a vendor who didn't take me seriously at all, but fixed my problem, than one who took my problem very seriously, but closed the ticket with the issue unfixed six months later, with half a dozen perfectly fine suggestions in the thread, and literally several books written about this exact topic (ie, fixing bad choices made in the past without breaking legacy support.)

If you take this seriously, why isn't it fixed? It isn't unfixable.

Buddy, I don't know you. All I know is I told someone that they could do work for me in Node because Node claimed first class Windows support, except I can't install d3 or lodash or moustache or facebook react or graphael, all because the javascript engine chose a path system that was well defined to be wrong before they got started, and are now blaming Microsoft for their own mistake.

Dude, no, this isn't a bug in Windows. You're trying to hack around things by using the Unicode API for something it wasn't intended for, then saying "well all Windows software should be expected to update itself to obscure unofficial APIs that don't even exist in most languages so that we don't have to make a replacement module api!"

Your staff are busily closing all of your customers' and potential customers' avenues for telling you that they are unhappy because the tool you made has an unnecessary defect.

Please just fix the path strategy of your module system, or add a successor path system without the limitation so that library vendors can self-migrate past the problem. It's not a lot to ask for a tool vendor that claims first class support to be expected fundamentally to work at all.

[isaacs]

For posterity, this was the resolution of the OP: #6960 (comment)

We are working on changing the way that npm lays modules out on the file system. Node is not the faulty technology here, it has nothing to do with package installation.