fs.readdir seems to sometimes wait forever

[mortie]

Hi. I recently decided to try to replace node.js with io.js. I encountered an issue with a project of mine, https://github.com/mortie/nouwell, where a program designed to do one task and then finish never finished.

I have narrowed it down to this fs.readdir:
https://github.com/mortie/nouwell/blob/d030b1c0c3a987e437bc2f74ac54c3c909d32202/js/database/index.js#L117

Adding a console.log before the statement, and another at the very top of the callback function, like so:

console.log("one");
fs.readdir(path.join(self.root, table), function(err, res)
{
    console.log("two");
    /* ... */
});

The first time the getFiles method is invoked, both console.log statements are invoked, but all the three subsequent times, only the first console.log seems to be executed, and the fs.readdir is seemingly waiting indefinitely.

I haven't been able to reproduce the issue outside of that one case, but due to the characteristics described above, and the fact that everything worked nicely with node.js, seems to indicate that it's an issue with io.js.

io.js version: v1.4.3
OS: Linux, Debian Wheezy (Linux gallifrey 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux)

[brendanashworth]

Is there an extraordinary amount of files in the directory you're reading?

[rvagg]

@mortie I fear that without being able to narrow this down to a simpler code sample that can be run by maintainers that this is going to remain unresolved because we cannot verify it and it may be something in your code that's doing this.

One thing that does occur to me here is that there could be a garbage collection interaction that's doing something new in io.js compared to older versions of V8 (have you tried this with Node 0.12 btw?). Try this: hoist gotDir out of the argument to readdir and put it back in by name only. Then, outside of the call to readdir, attach it to the cb argument as a property--this should ensure it's kept as a reference and not garbage collected. Something like:

function getFiles(table, cb) {
var self = this;

function gotDir(err, files) {
  // ...
}
cb.gotDir = gotDir
fs.readdir(path.join(self.root, table), gotDir)

Of course this doesn't make sense for standard Node (or JS) programming but I've encountered similar errors with my native addon code where I've neglected to keep a "persistent" reference to callback functions passed in to the C++ and when they also don't have a reference in the JavaScript they become GC candidates. I find it difficult to imagine we have a problem like this in core but it may be worth exploring the possibility.

[brendanashworth]

Also, your entire event loop may be held up. After launching the fs.readdir function (not in the callback), could you try the following? If it logs "should log" then this isn't the issue.

process.nextTick(console.log.bind(console, 'should log'));

[Fishrock123]

ping @mortie

[tellnes]

This issue seems dead. Please feel free to reopen if someone has more information.