Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

request: add option to call fsync() when closing files #49886

Closed
cjihrig opened this issue Sep 26, 2023 · 8 comments
Closed

request: add option to call fsync() when closing files #49886

cjihrig opened this issue Sep 26, 2023 · 8 comments
Labels
feature request Issues that request new features to be added to Node.js. fs Issues and PRs related to the fs subsystem / file system.

Comments

@cjihrig
Copy link
Contributor

cjihrig commented Sep 26, 2023

What is the problem this feature will solve?

I believe it is possible that Node can close a file and then try to interact with the same file before all of the data has been written to disk. According to the docs for close(2) this can be fixed by calling fsync().

What is the feature you are proposing to solve the problem?

Add an option to the appropriate fs functions (close(), writeFile(), etc.) to call fsync() automatically.

What alternatives have you considered?

Flushing the data "by hand."

@cjihrig cjihrig added the feature request Issues that request new features to be added to Node.js. label Sep 26, 2023
@aduh95 aduh95 added the fs Issues and PRs related to the fs subsystem / file system. label Sep 26, 2023
@aduh95
Copy link
Contributor

aduh95 commented Sep 26, 2023

/cc @nodejs/fs

@bnoordhuis
Copy link
Member

For fs.writeFile() that makes sense, it's a sequence of system calls.

For fs.close() it makes less sense, IMO. Just call fs.fsync() before fs.close().

See also #28513 about interleaving fs.fsync() and fs.WriteStream.

@mcollina
Copy link
Member

I think adding an option to fs.WriteStream would be helpful too.

@bnoordhuis as a side note, I have been experiencing significant read-after-write issues when using Node.js fs APIs since the move to IO_URING. It's too fast and the filesystem is not flushed.

@benjamingr
Copy link
Member

I think writeFile and fs.WriteStream makes sense (especially writeFile)

@cjihrig
Copy link
Contributor Author

cjihrig commented Oct 1, 2023

In addition to writeFile() and fs.WriteStream, there are a few other APIs where this could make sense:

  • appendFile() - calls writeFile() under the hood with a modified copy of its options. This one should Just Work™, but should include tests as well.
  • truncate() - opens the file, calls ftruncate(), and then closes the file.
  • lchmod() - opens the file, calls fchmod(), and then closes the file. I'm less sure if it is necessary here.
  • copyFile() - this one is tricky because it is implemented several different ways in libuv and Node never has access to the file descriptor.

cjihrig added a commit to cjihrig/node that referenced this issue Oct 2, 2023
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: nodejs#49886
cjihrig added a commit to cjihrig/node that referenced this issue Oct 2, 2023
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: nodejs#49886
nodejs-github-bot pushed a commit that referenced this issue Oct 4, 2023
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: #49886
PR-URL: #50009
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
cjihrig added a commit to cjihrig/node that referenced this issue Oct 8, 2023
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: nodejs#49886
cjihrig added a commit to cjihrig/node that referenced this issue Oct 8, 2023
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: nodejs#49886
cjihrig added a commit to cjihrig/node that referenced this issue Oct 8, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in nodejs#50009, but this makes it more official.

Refs: nodejs#49886
Refs: nodejs#50009
cjihrig added a commit to cjihrig/node that referenced this issue Oct 8, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in nodejs#50009, but this makes it more official.

Refs: nodejs#49886
Refs: nodejs#50009
nodejs-github-bot pushed a commit that referenced this issue Oct 11, 2023
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: #49886
PR-URL: #50093
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
nodejs-github-bot pushed a commit that referenced this issue Oct 18, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in #50009, but this makes it more official.

Refs: #49886
Refs: #50009
PR-URL: #50095
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
targos pushed a commit that referenced this issue Oct 23, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in #50009, but this makes it more official.

Refs: #49886
Refs: #50009
PR-URL: #50095
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
@colemickens
Copy link

I think writeFile and fs.WriteStream makes sense (especially writeFile)

I came here because I think I'm having issues with a fs.WriteStream and syncing before closing, but it does seem that fs.createWriteStream takes options that specify there should be flushing before closing. https://nodejs.org/api/fs.html#filehandlecreatewritestreamoptions

(I'm partly posting this to check my own assumptions here, 🙏 )

@mcollina
Copy link
Member

I think writeFile and fs.WriteStream makes sense (especially writeFile)

I came here because I think I'm having issues with a fs.WriteStream and syncing before closing, but it does seem that fs.createWriteStream takes options that specify there should be flushing before closing. https://nodejs.org/api/fs.html#filehandlecreatewritestreamoptions

(I'm partly posting this to check my own assumptions here, 🙏 )

Could you better articulate your question?

@colemickens
Copy link

Sorry, I didn't realize that the new option in node 21 was a result of this thread. There's no remaining question, now.

alexfernandez pushed a commit to alexfernandez/node that referenced this issue Nov 1, 2023
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: nodejs#49886
PR-URL: nodejs#50009
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
alexfernandez pushed a commit to alexfernandez/node that referenced this issue Nov 1, 2023
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: nodejs#49886
PR-URL: nodejs#50093
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
alexfernandez pushed a commit to alexfernandez/node that referenced this issue Nov 1, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in nodejs#50009, but this makes it more official.

Refs: nodejs#49886
Refs: nodejs#50009
PR-URL: nodejs#50095
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
targos pushed a commit that referenced this issue Nov 11, 2023
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: #49886
PR-URL: #50009
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
targos pushed a commit that referenced this issue Nov 11, 2023
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: #49886
PR-URL: #50093
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
targos pushed a commit that referenced this issue Nov 11, 2023
This commit adds documentation and tests for the 'flush' option
of the fs.appendFile family of functions. Technically, support
was indirectly added in #50009, but this makes it more official.

Refs: #49886
Refs: #50009
PR-URL: #50095
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
@cjihrig cjihrig closed this as completed Nov 19, 2023
debadree25 pushed a commit to debadree25/node that referenced this issue Apr 15, 2024
This commit adds a 'flush' option to the fs.writeFile family of
functions.

Refs: nodejs#49886
PR-URL: nodejs#50009
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
debadree25 pushed a commit to debadree25/node that referenced this issue Apr 15, 2024
This commit adds a 'flush' option to the createWriteStream()
family of functions.

Refs: nodejs#49886
PR-URL: nodejs#50093
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Issues that request new features to be added to Node.js. fs Issues and PRs related to the fs subsystem / file system.
Projects
None yet
Development

No branches or pull requests

6 participants