add an option to not delete node_modules

[emexelem]

I'm managing CI servers with a large number js projects and i'm looking to optimize the duration and the IO spent on npm install, which represents a significant part of the total amount of the build.

I've been looking into npm ci as a replacement of npm install as soon as i heard of it, but was disappointed by the fact that npm ci deletes the whole node_modules, which makes it less efficient than npm install when a node_modules set up by a previous CI build is available.

I just discovered this cipm and give it a try. Here's the results of the tests I made on one js project installing 1200 modules. I measured the real user time, cpu time, system cpu time, and read and written MB by the process:

Main points:

• cipm is much faster (36" vs 120") and less IO intensive (especially on writes) than npm install on fresh builds, when there's no node_modules available
• on subsequent builds, when node_modules is already available, npm install is much faster (19" vs 162" ! ) and less IO intensive (especially on reads) than cipm

The fact that cipm, as npm ci, deletes the whole node_modules, has a huge impact on duration and IO spent.
The best combination for now would be to use cipm during the initial build when there's no node_modules, and then use npm install.
But i think cipm could be faster than npm install if it did not delete node_modules.

Would it be possible to add a parameter to not delete node_modules?
As most of the files there won't change, there must be a smarter way to do this than deleting and reinstalling everything.

[emexelem]

Hey,
Also included results with npm ci

[seanmcilvenna]

I agree, this would be useful.
I also found this issue: npm/npm#20104
But the npm/npm repo has been archived...
I don't think the history/interest of that issue should be lost, though. Would like to see this --no-delete option in the near future.

[emexelem]

i also raised the same issue in the npm issue tracker: https://npm.community/t/running-npm-ci-without-deleting-old-node-modules-folder-20104/3152/7
I'll pick the first tool implementing this :)

[seanmcilvenna]

I don't believe this should be closed. I believe the intent behind this ticket is to improve performance by not deleting the node_modules or the contents of node_modules. Rather, npm ci should be intelligent enough to scan what is already installed in the node_modules directory and make a determination of whether the dependencies needs to be changed, removed, added to the node_modules directory. Removing 1000 dependency directories every time you run npm ci is not performance-friendly.

[emexelem]

Fully agree with @seanmcilvenna.
The commit you referenced @claudiahdz preserves the node_modules directory itself but deletes its content, this does not address at all the problem raised here.
npm ci should look at package-lock.json and then at the content of node_modules to computes a diff, a bit as rsync, and then only performs what's needed (installing new modules, updating versions, or removing removed modules).
If package-lock.json has not changed since last build, then npm ci should not do anything in node_modules.

[claudiahdz]

It is a fact that npm ci is only faster when node_modules does not exist. Its intention is to be more strict and allow a clean install of dependencies on CI environments, hence the deletion of the folder. Its performance benefit is from the fact that it doesn't have to look in package.json, build a tree, and compute whether that tree is up to date or not.

Having said so, the npm CLI team is currently working on v7 which includes a big refactor that aims to improve all aspects of the CLI (including making npm install faster and more reliable). If you're interested check: https://github.com/npm/arborist.

[stefanpl]

@claudiahdz I found here after seeing your post on npm.community.
There's two things:

• the roadmap does not explicitly talk about this feature (option for ci to not delete node_modules) being a part of v7. Will it definitely be? Will it be in 7.0.0?
• roadmap says: "Initial release will almost certainly be this year" … any updates on this? ;)

Answers to these questions would make the decision to wait/hack-around a lot easier for those of us struggling with this. Thank you so much for your support!