Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(node/process): null is not returned when reaching end-of-file in stdin #3113

Merged
merged 24 commits into from
Feb 3, 2023

Conversation

PolarETech
Copy link
Contributor

@PolarETech PolarETech commented Jan 15, 2023

Fixes: #2974

Currently, the highWaterMark value of Readable for process.stdin is fixed at 0. Therefore, when piping a file, null is not returned after the file is read to the end, and the end event is not emitted.
This PR will fix this behavior by setting the appropriate highWaterMark value based on the type of stdin.

TODO
Actual highWaterMark values in Node.js.

stdin Linux/Mac Windows Related streams
TTY
  • 0
  • 0
tty.ReadStream (Duplex)
FILE
  • 65536
  • 65536
fs.ReadStream (Readable)
PIPE
  • 16384
  • 16384 (*1)
net.Socket (Duplex)
TCP
  • 16384 (*1)
- (will not happen) net.Socket (Duplex)
null
  • 65536
  • 65536 (*1)
fs.ReadStream (Readable)
other
  • 16384
  • 16384
unresolved (*2)
Readable

(*1) The accuracy of type identification is insufficient and needs further improvement.
(*2) Non-console Windows applications are assumed (ref), but I couldn't find a way to get this state in Deno. The value in Node.js was checked in an electron app.

Will not be addressed in this PR

  • Implementation of tty.ReadStream
    Use Duplex instead
  • Implementation of net.Socket with fd option
    Use Duplex instead
  • Implementation of guessHandleType() in "node/internal_binding/util.ts"

Refs
The highWaterMark value to be set is based on actual Node.js behavior and some source code.
https://github.com/nodejs/node/blob/v18.12.1/lib/internal/bootstrap/switches/is_main_thread.js#L189
https://github.com/nodejs/node/blob/v18.12.1/src/node_util.cc#L257
https://github.com/nodejs/node/blob/v18.12.1/deps/uv/src/unix/tty.c#L333
https://github.com/nodejs/node/blob/v18.12.1/deps/uv/src/win/handle.c#L31
https://github.com/nodejs/node/blob/v18.12.1/lib/tty.js#L60
https://github.com/nodejs/node/blob/v18.12.1/lib/internal/fs/streams.js#L155
https://github.com/nodejs/node/blob/v18.12.1/lib/net.js#L329
https://github.com/nodejs/readable-stream/blob/v4.2.0/lib/internal/streams/state.js#L11-L13

Node test suite
"pseudo-tty/test-readable-tty-keepalive.js" test seems to be related to this change but does not pass the test. The data is not received as expected in process.stdin.on('readable', ...).
The test has not been added because it appears to be failing due to factors other than this change, and it may take time to investigate. Perhaps some kind of PTY is needed in addition.

Memo
Some issues with the node compat layer have been found while working on this PR.

  • stdin of stdio option is ignored in child_process.spawnSync() (src)
  • input option is ignored in child_process.spawnSync() (src)

@PolarETech PolarETech marked this pull request as ready for review January 15, 2023 11:36
@PolarETech PolarETech requested a review from kt3k as a code owner January 15, 2023 11:36
node/_process/streams.mjs Outdated Show resolved Hide resolved
@kt3k
Copy link
Member

kt3k commented Jan 15, 2023

Can you add some custom test case in node/process_test.ts which exercise the case this PR fixes?

Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
@PolarETech
Copy link
Contributor Author

Two unit test cases have been added.
https://github.com/denoland/deno_std/pull/3113/files#diff-f27eacddc948ff5c5ce7c851fccf64c00f313b8edc28862b5066c211e2b14d72

  1. process.stdin readable with a TTY
  2. process.stdin readable with piping a file

However, Github Actions runs the tests without a TTY. So, I have implemented the first unit test case to be ignored on CI, because it cannot run the test as expected. Any solutions to this situation would be appreciated.
I have confirmed the test passes on local Linux, Mac, and Windows.

@PolarETech PolarETech marked this pull request as ready for review January 16, 2023 11:46
async fn() {
const expected = ["65536", "foo", "bar", "null", "end"];
const scriptPath = "./node/testdata/process_stdin.ts";
const file = await Deno.readFile(`./node/testdata/process_stdin_dummy.txt`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can replace this with inline Uint8Array like: new TextEncoder().encode("foo\nbar")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test was divided into two tests, one for file and one for pipe. And I used inline Uint8Array in the test for pipe.
https://github.com/denoland/deno_std/pull/3113/files#diff-f27eacddc948ff5c5ce7c851fccf64c00f313b8edc28862b5066c211e2b14d72R402

Deno.test({
name: "process.stdin readable with piping a file",
async fn() {
const expected = ["65536", "foo", "bar", "null", "end"];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In node.js, the first number becomes 16384. Is this difference intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My confirmation was lacking.
As I mentioned in the issue, the highWaterMark is 65536 with piping a file, But when running the code in child_process with piping stdin, the highWaterMark is a different value.

  • tty(tty.ReadStream): 0
  • piping files(fs.ReadStream): 65536
  • piping stdin(Stream?): 16384

I'll look into it further.
Thank you for pointing this out.

@PolarETech PolarETech marked this pull request as draft January 17, 2023 10:40
node/_process/streams.mjs Outdated Show resolved Hide resolved
node/_process/streams.mjs Outdated Show resolved Hide resolved
@PolarETech
Copy link
Contributor Author

The stdin implementation has been changed to be more in line with Node.js.
Some TODOs remain, but they seem difficult to solve in a short time and I would like to be allowed to postpone them.

@PolarETech PolarETech requested a review from kt3k February 2, 2023 14:45
@PolarETech PolarETech marked this pull request as ready for review February 2, 2023 14:45
Copy link
Member

@kt3k kt3k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work! Thanks! @PolarETech

@kt3k kt3k merged commit e6592e4 into denoland:main Feb 3, 2023
@PolarETech PolarETech deleted the issue-2974 branch February 3, 2023 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Streaming stdin using process.stdin does not work
3 participants