Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the chunk boundary (node:stream:createReadStream) #3853

Merged
merged 3 commits into from
Jul 28, 2023

Conversation

Hanaasagi
Copy link
Collaborator

@Hanaasagi Hanaasagi commented Jul 28, 2023

Close: #3668

What does this PR do?

  • Documentation or TypeScript types (it's okay to leave the rest blank in this case)
  • Code changes

How did you verify your code works?

Test Code:

const fs = require("node:fs");
const s = require("node:stream");

const path = "./test.txt";
fs.writeFileSync(path, "12345");

const stream = fs.createReadStream(path, { start: 0, end: 4 });

const writable = new s.Writable({
  write(chunk, encoding, callback) {
    console.log(chunk);
  },
});

stream.pipe(writable);

2023-07-28_13-03

Checklist

If JavaScript/TypeScript modules or builtins changed:

  • I ran make js and committed the transpiled changes
  • I or my editor ran Prettier on the changed files (or I ran bun fmt)
  • I included a test for the new code, or an existing test covers it

If Zig files changed:

  • I checked the lifetime of memory allocated to verify it's (1) freed and (2) only freed when it should be
  • I or my editor ran zig fmt on the changed files
  • I included a test for the new code, or an existing test covers it
  • JSValue used outside outside of the stack is either wrapped in a JSC.Strong or is JSValueProtect'ed

If new methods, getters, or setters were added to a publicly exposed class:

  • I added TypeScript types for the new methods, getters, or setters

If dependencies in tests changed:

  • I made sure that specific versions of dependencies are used instead of ranged or tagged versions

If functions were added to exports.zig or bindings.zig

  • I ran make headers to regenerate the C header file

If *.classes.ts files were added or changed:

  • I ran make codegen to regenerate the C++ and Zig code

If a new builtin ESM/CJS module was added:

  • I updated Aliases in module_loader.zig to include the new module
  • I added a test that imports the module
  • I added a test that require() the module

@Jarred-Sumner
Copy link
Collaborator

i'm surprised there's an off-by-one error here but it also is consistent with the bug reports show.

you're sure it's not the += thing in http.write?

@Hanaasagi
Copy link
Collaborator Author

I need some time to test this issue, but at least this is not related to http.write. You can run this code, and compare the output with node.js.

const fs = require("node:fs");
const s = require("node:stream");

const path = "./test.txt";
fs.writeFileSync(path, "12345");

const stream = fs.createReadStream(path, { start: 0, end: 4 });

const writable = new s.Writable({
  write(chunk, encoding, callback) {
    console.log(chunk);
  },
});

stream.pipe(writable);

This won't execute any code related to http, just used stream.pip.

@Hanaasagi Hanaasagi marked this pull request as ready for review July 28, 2023 11:56
@Hanaasagi
Copy link
Collaborator Author

Hi, here's an explanation of why the createReadStream should be fixed.

As mentioned in issue #3668, the server-static package of express depends on send, which uses createReadStream to create read streams.

https://github.com/pillarjs/send/blob/b69cbb3dc4c09c37917d08a4c13fcd1bac97ade5/index.js#L691-L704
https://github.com/pillarjs/send/blob/b69cbb3dc4c09c37917d08a4c13fcd1bac97ade5/index.js#L777-L793

According to the Node.js documentation, end refers to the offset, not the size.

Link: https://nodejs.org/api/fs.html#fscreatereadstreampath-options

options can include start and end values to read a range of bytes from the file instead of the entire file. Both start and end are inclusive and start counting at 0, allowed values are in the [0, Number.MAX_SAFE_INTEGER] range.

@Hanaasagi Hanaasagi changed the title [WIP]fix the chunk boundary (node:stream:createReadStream) fix the chunk boundary (node:stream:createReadStream) Jul 28, 2023
@Jarred-Sumner Jarred-Sumner merged commit 9b91e3c into oven-sh:main Jul 28, 2023
@Jarred-Sumner
Copy link
Collaborator

Thank you

trnxdev pushed a commit to trnxdev/bun that referenced this pull request Aug 9, 2023
* fix the slice boundary.
Close: oven-sh#3668

* Add more boundary test case.

* fix end is 0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Missing last byte when serving static files
2 participants