EPERM when renaming files on Windows

[achingbrain]

• Version: 10.16.3
• Platform: Windows 10
• Subsystem: fs

The docs for fs.rename(oldPath, newPath, callback) say:

In the case that newPath already exists, it will be overwritten.

This happens on Linux/Mac OS X but doesn't appear to happen with Windows. If you run the following script:

const { tmpdir } = require('os')
const { join } = require('path')
const { writeFile, rename } = require('fs')

const dest = join(tmpdir(), 'dest.txt')
const file = Buffer.allocUnsafe(1024 * 1024) // 1MB
const count = 100

for (let i = 0; i < count; i++) {
  writeFile(`${dest}-${i}.tmp`, file, (err) => {
    if (err) {
      console.error('Could not write')
      throw err
    }

    rename(`${dest}-${i}.tmp`, dest, (err) => {
      if (err) {
        console.error('Could not rename')
        throw err
      }
    })
  })
}

You will see:

C:\Users\me> node test.js
Could not rename
C:\Users\me\test.js:21
        throw err
        ^

Error: EPERM: operation not permitted, rename 'C:\Users\me\AppData\Local\Temp\dest.txt-0.tmp' -> 'C:\Users\me\AppData\Local\Temp\dest.txt'

The same script runs to completion on Linux & Mac OS X without an error.

Setting count to a low value (e.g. 2) doesn't trigger the bug for me, so if it works for you on Windows, your computer is likely faster than mine - try setting it to something unreasonable like 1000.

This could be what was happening in #6335 and appears at least tangentially related to #21957

[bnoordhuis]

See discussion in libuv/libuv#1981 - which was subsequently reverted in response to libuv/libuv#2098.

Long story short: this is not something that's really under Node's or libuv's control and the best we could do is try to work around it by means of a grace period.

That's what modules like graceful-fs do, it arguably has no place in Node.js itself.

[vtjnash]

I noticed that Windows 10 recently added some features that perhaps might help here. In particular, FILE_RENAME_POSIX_SEMANTICS and FILE_RENAME_POSIX_SEMANTICS. Posting here (instead of libuv) to get more eyeballs on this and to ask whether these are flags that libuv should try to use.

[kriskowal]

For reference, this problem was tackled in the Go community by making rename work consistently between POSIX and Windows. Python encountered the same foible. golang/go#8914

[mbargiel]

Integrating FILE_RENAME_POSIX_SEMANTICS sounds like a good idea to me (if that's any feasible, that is). Retrying via a separate library is not always an option: what if the handle is not held temporarily, but by an actual program executing from inside the folder being renamed and which will not shut down in a second or two? The renamer might conceptually have the authority to pull the proverbial rug from under the executable, so it would be important to make it possible to allow this to happen. That would enable identical application codepaths in the JavaScript code rather than force OS-dependent handling, enabling more portability.

Barring that, the only alternative I see is for the JavaScript application code to have a code path detecting this specific failure on Windows, then using low-level Windows APIs to try and detect the software holding the handles ... and terminate them/forcibly close the handles/call Windows' own rename API with whatever flags needed for the rename to work in that situation. In any case, it would make for a comparatively much more complex workaround for something that could be rather simple to implement in NodeJS's fs.rename API - exposing this via an option, if the unforeseen side-effects of the change make it too risky to change the default rename behaviour altogether.

[Saifullah-dev]

C:\Users\me> node test.js
Could not rename
C:\Users\me\test.js:21
        throw err
        ^

Error: EPERM: operation not permitted, rename 'C:\Users\me\AppData\Local\Temp\dest.txt-0.tmp' -> 'C:\Users\me\AppData\Local\Temp\dest.txt'

Use node server.js instead of nodemon server.js as it locks files as a listener, see Stackoverflow Answer.

Also note that Windows Policy doesn't allow you to rename the file that is open in any other program. So, close before renaming.