Raise the maximum open files limit on Windows

[TheStormN]

What is the problem this feature will solve?

The maximum open files limit is increased to maximum on POSIX platforms here

node/src/node.cc

Line 528 in 2a4452a

// Raise the open file descriptor limit.

However, similar functionality is missing for Windows which causes issues with error "EMFILE: too many open files", specially when using webpack.

What is the feature you are proposing to solve the problem?

Use the Windows CRT API to increase the limit. The function is called _setmaxstdio - https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170

Different versions of Windows support different limits, the default is 512 files with maximum possible number of 8192. My suggestion would be to use loop for probing the maximum possible value, similar to what is currently done for POSIX. I.e. something like that:

static constexpr int MIN_OPEN_FILES_LIMIT = 512;
static constexpr int MAX_OPEN_FILES_LIMIT = 8192;
static constexpr int STEP_OPEN_FILES_LIMIT = 512;
for (int i = MAX_OPEN_FILES_LIMIT; i > MIN_OPEN_FILES_LIMIT; i -= STEP_OPEN_FILES_LIMIT)
{
    if (_setmaxstdio(i) != -1)
        break;
}

What alternatives have you considered?

Unfortunately, it seems there are no alternatives which can raise the limit for process on OS level (like ulimit on some *nix OSes). The process itself have to request it.

[bnoordhuis]

Pull request welcome, I'd say. I'm curious what exactly causes those "too many open files" errors because node/libuv by and large doesn't use the CRT.

[vtjnash]

It is libuv v2 that doesn't use the CRT, and thus has no upper limit on files. libuv v1.x uses it heavily for files and pipes, and thus has an upper limit of 2k open files (or 8k if building with MSVC 2022)

[github-actions]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[github-actions]

There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[thebanjomatic]

I'm still frequently experiencing this problem when using vitest as vitest relies on node's native dynamic import implementation for loading es-modules and lots of concurrent imports in different worker threads can lead to exhaustion of the file-descriptor pool. However, I have implemented the above suggestion in the description and found it to be unnecessary as that API increases the number of concurrent stream i/o operations, but the low-level i/o we are using already supports 8192 descriptors. In my case after we have 8192 files open concurrently, it returns -1 and then the process crashes with EMFILE: too many files so another solution would be needed.

I'm able to reproduce this problem in node by itself by just using node to import a large internal library (with about 1000 .js sub-import) in a number of worker threads. It works for 11 concurrent workers, and the breaks at 12 when the file descriptors are exhausted.

It seems like libuv v2 would solve this problem by using the windows api directly instead of using the c runtime for its implementation, but from what I can tell v2 is not yet released and it may be a long time before it gets adopted in a node release.

If these were just regular fs module readFile / openFile operations, we could use something like graceful-fs on the app or library side to handle the EMFILES error and wait until more files are closed before retrying, but since this is occurring inside of a dynamic import, monkey patching fs won't work here.

Would it make sense to apply that graceful retry logic at a lower level within node by wrapping uv_fs_open and uv_fs_close with a retry queue?