Intermittently losing an 'add' file event with nodefs-handler.js

[bmathews]

On Windows, I'm using chokidar to monitor a single flat folder that is receiving photos being taken by a camera, and about once every 20 or 30 photos an 'add' event ends up lost.

I've added a whole bunch of logging to chokidar and node's fs.js module, and I think I've narrowed down the problem to receiving the fs event while chokidar is running readdirp. It's possible that my understanding of how chokidar is working is incorrect, so I wanted to post this to get a second opinion before I look into a fix.

Inside of nodefs-handler.js _handleDir read function, there's some code that bails if readdir is already being called:

      var throttler = this._throttle('readdir', directory, 1000);
      if (!throttler) return;

I believe this causes chokidar to essentially ignore any events that're happening while readdir is still running. On my windows machine with this particular folder of images, readdirp is taking about 30ms to run. My current understanding is that the fs event that would indicate my photo has been added is occurring within that ~30ms window and being ignored, and that readdirp is for whatever reason not returning it in its list of files. I can't imagine that readdirp can guarantee that its list of files is correct if they're changing out from underneath it mid-call.

If all of that sounds plausible to someone who knows the code pretty well, then I'd be happy to look into fixing it.

Thanks!

[es128]

Just to make sure, have you inspected output to the raw event to ensure the fs.watch() subsystem is emitting events for the files that were missed?

If so, then your finding appears feasible, and the solution may be as simple as tweaking the throttle methodology. Perhaps it should be more of a debounce with both leading and trailing invocation.

[bmathews]

I did verify that an fs.watch event is emitted for the file that was missed, yes.

Debouncing with a leading/trailing invocation does seem like the right solution. I'm not seeing that as a particularly "simple" task here, but it might just be because I don't understand the code as well as you. Does that seem like something you could (and would be willing to) put together quickly? Or should I see what I can come up with?

[srguiwiz]

Appreciate fix done by @bmathews , taking initiative. Flaw with that fix discussed at #723. Sorry if I didn't pick more careful words, but I do appreciate work done in #690.

If you can review that #723 we might have a better fix in 2.0.4.

[srguiwiz]

While making test cases for PR #723 discussed and found out the suspected flaw in PR #690 wasn't really happening.

There are still differences between the two fixes.