npm cache clean doesn't clean hard-source cache

[JGJP]

I've been getting some errors that look like EPERM: cannot rename cache/hard-source/somefile when I try to build. It's a problem with the cache, but running npm cache clean doesn't delete node_modules\.cache\hard-source. The problem is resolved when I delete this folder manually, but it would be nice if we could clear it with npm cache clean, especially for my junior devs.

[JonWallsten]

I get the issue even if I clean the folder with rimraf before build. Although not all the times. Every now and then the build succeeds. Mostly failing though.

[mzgoddard]

Cache renaming happens when the cache compacts itself. This is probably a timing issue while compacting. Renaming the smaller cache copy may be happening too soon after deleting the larger original cache.

---

The cache has to write all the built modules to disk the first time webpack builds for a new cache. That time serializing and writing everything is a portion of the extra time HardSource uses on the first build. Following builds only write changes greatly reducing the time needed to write out to disk. To keep the cache from growing too large, it'll compact the cache once it reaches a certain size larger than the cache needs. It writes a second copy of the cache without the old entries, removes the old cache and renames the new copy to the old copies position.

[mzgoddard]

It'd be cool to be able to use npm cache clean to remove the current hard-source cache. But that isn't what that command means. It cleans the npm download cache meant to speed up installs, and I don't think there are any hooks available to connect to it. The default cache location is just an idiom other projects use. hard-source uses that idiom, for the same reason other packages do, since the cache will be deleted if you delete and reinstall npm modules, as well it makes it easier not to accidentally add it to a source control repo, publish it, etc.

I guess we could add a hard-source helper binary. That'd be great to clean the cache or get a list of caches, delete caches by date (like caches older than a week or two), or stats or something.

I'm not sure how to highlight that tool though to help in situations like this.

[mzgoddard]

Oh, to be clear I'll add a small timeout between the cache delete and rename during compacting in a patch release.

Thanks @JGJP for opening the issue and @JonWallsten for the additional info.

[mzgoddard]

@JonWallsten's added info in #260 makes it clear to me that the more likely culprit is the atomic json write in the cache behaviour. To echo #260 (comment), I'll add some retry behaviour and soft failure if it keeps happening so at least webpack can finish.

[mzgoddard]

@JGJP @JonWallsten if you have any thoughts on a hard-source helper binary for clearing the cache, etc. I'd really appreciate y'all sharing those with me.

[JonWallsten]

@mzgoddard I don't mind having to clean it manually. I do it as a precaution before I start a new build, or the watch, to make sure I start with a clean slate. I'm cleaning other stuff as well, so it's no extra work. Besides, there are no "standard" cleaning command to connect it to anyways, right?

[JGJP]

@mzgoddard Thanks for your replies. I don't think I have the expertise to comment on making a binary, I'll probably just end up running a Powershell function to delete the cache before crucial builds just in case.

[JonWallsten]

@JGJP Use the rimraf package, then you can add it as a script in your package.json.

[mzgoddard]

@JGJP I mean binary like the bin field in package.json of some packages like rimraf (https://github.com/isaacs/rimraf/blob/master/package.json#L12) and webpack. Like @JonWallsten mentions with rimraf installed in a project an npm script can refer to it to run it. You can also modify your environment's path and run them directly from your shell (How global npm installs give you access to those commands).

[JGJP]

Oh ok I get it now, thanks!