Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a bug where errors could leak on extremely large stream chunks #386

Closed

Conversation

haggholm
Copy link
Contributor

I don't expect a lot of people are going to see this, but we have encountered an issue where an error is thrown as an unhandled exception:

Cannot create a string longer than 0x1fffffe8 characters

This happens here:

    stringifier.on('end', function(){
      callback(undefined, chunks.join(''));
    });

Obviously, if chunks.join() throws an error, it is leaked rather than passed to the callback. I suppose you don't normally expect chunks.join() to fail, but it is in fact posssible.

I'm not sure what style you prefer for this sort of thing; this ought to work and be proof to double-invoking callbacks, but feel free to edit as needed or desired, obviously.

@wdavidw
Copy link
Member

wdavidw commented Apr 14, 2023

Thank you for reporting and preparing a fix, we will need a test case to reproduce the issue, do you think you could prepare one ?

@haggholm
Copy link
Contributor Author

Maybe you can adapt something from something along these lines? Throws an unhandled exception before my fix, returns it to the callback with it. Of course it's kind of annoying because it's takes up a bunch of CPU and memory, but, well, it only happens with huge strings.

const {stringify} = require("csv-stringify");
const {randomBytes} = require("crypto");

stringify(Array.from({length: 200_000}).map(
	() => Array.from({length: 100}).map(
		() => "ABCDEFGHIJKLMNOPQRSTUVXYZ0123456789")), (err, res) => {
	if (err) {
		console.error("Failed with error:", err);
	} else {
		console.info("Content length:", res.length);
	}
});

@wdavidw wdavidw closed this in 1d500ed Apr 16, 2023
@wdavidw
Copy link
Member

wdavidw commented Apr 16, 2023

Thank you!

salceson added a commit to evidenceprime/node-csv that referenced this pull request Sep 22, 2023
* chore: latest dependencies

* fix: uncaught errors with large stream chunks (fix adaltas#386)

* chore(release): publish

 - csv-demo-browser@0.1.6
 - csv-demo-cjs@0.2.4
 - csv-demo-eslint@0.1.10
 - csv-demo-esm@0.0.18
 - csv-issues-cjs@0.1.5
 - csv-issues-esm@0.0.9
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.4
 - csv-demo-ts-module-node16@0.2.4
 - csv-demo-webpack-ts@0.1.6
 - csv-demo-webpack@0.1.8
 - csv-generate@4.2.3
 - csv-parse@5.3.7
 - csv-stringify@6.3.1
 - csv@6.2.9
 - stream-transform@3.2.3

* test(csv-stringify): fix legacy

* chore(release): publish

 - csv-demo-browser@0.1.7
 - csv-demo-cjs@0.2.5
 - csv-demo-eslint@0.1.11
 - csv-demo-esm@0.0.19
 - csv-issues-cjs@0.1.6
 - csv-issues-esm@0.0.10
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.5
 - csv-demo-ts-module-node16@0.2.5
 - csv-demo-webpack-ts@0.1.7
 - csv-demo-webpack@0.1.9
 - csv-generate@4.2.4
 - csv-parse@5.3.8
 - csv-stringify@6.3.2
 - csv@6.2.10
 - stream-transform@3.2.4

* build: remove trailing slash in home url

* chore: latest dependencies

* fix(csv): fixed CJS types under modern `modernResolution` options (adaltas#388)

* fix(csv): remove ts files in cjs dist

* chore(release): publish

 - csv-demo-browser@0.1.8
 - csv-demo-cjs@0.2.6
 - csv-demo-eslint@0.1.12
 - csv-demo-esm@0.0.20
 - csv-issues-cjs@0.1.7
 - csv-issues-esm@0.0.11
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.6
 - csv-demo-ts-module-node16@0.2.6
 - csv-demo-webpack-ts@0.1.8
 - csv-demo-webpack@0.1.10
 - csv-generate@4.2.5
 - csv-parse@5.3.9
 - csv-stringify@6.3.3
 - csv@6.2.11
 - stream-transform@3.2.5

* docs: minor upercase modification

* chore: latest dependencies

* chore(release): publish

 - csv-demo-browser@0.1.9
 - csv-demo-cjs@0.2.7
 - csv-demo-eslint@0.1.13
 - csv-demo-esm@0.0.21
 - csv-issues-cjs@0.1.8
 - csv-issues-esm@0.0.12
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.7
 - csv-demo-ts-module-node16@0.2.7
 - csv-demo-webpack-ts@0.1.9
 - csv-demo-webpack@0.1.11
 - csv-generate@4.2.6
 - csv-parse@5.3.10
 - csv-stringify@6.3.4
 - csv@6.2.12
 - stream-transform@3.2.6

* feat: add unicode chars to formula escape (adaltas#387)

* fix(csv-stringify): use switch in formula escaping

* fix(csv-stringify): add unicode character equivalents in formula sanitization

* chore: update tests

* docs(csv-stringify): escape formulas references

* chore(release): publish

 - csv-demo-browser@0.1.10
 - csv-demo-cjs@0.2.8
 - csv-demo-eslint@0.1.14
 - csv-demo-esm@0.0.22
 - csv-issues-cjs@0.1.9
 - csv-issues-esm@0.0.13
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.8
 - csv-demo-ts-module-node16@0.2.8
 - csv-demo-webpack-ts@0.1.10
 - csv-demo-webpack@0.1.12
 - csv-stringify@6.4.0
 - csv@6.3.0

* feat(csv-parse): add `columns` property in `Info` object type (adaltas#390)

* fix(ts): Add `columns` property in `Info` object type

* Add disabled options to columns type

* build(csv-parse): build and write test after info ts definition

* chore(release): publish

 - csv-demo-browser@0.1.11
 - csv-demo-cjs@0.2.9
 - csv-demo-esm@0.0.23
 - csv-issues-cjs@0.1.10
 - csv-issues-esm@0.0.14
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.9
 - csv-demo-ts-module-node16@0.2.9
 - csv-demo-webpack-ts@0.1.11
 - csv-demo-webpack@0.1.13
 - csv-parse@5.4.0
 - csv@6.3.1

* docs: update build badge urls

* docs(csv-generate): comment indentation in samples

* refactor(csv-issues-cjs): code format

* refactor(csv-issues-cjs): remove unused arguments

* test(csv-issues-cjs): fix stdout maxBuffer length exceeded

* test(csv-issues-esm): use spawn instead of exec

* fix: commonjs types, run tsc and lint to validate changes (adaltas#397)

* fix: types weren't working for commonjs. Run tsc and lint to validate changes

* chore: needs to work on linux and BSD

* chore: latest dependencies

* chore(release): publish

 - csv-demo-browser@0.1.12
 - csv-demo-cjs@0.2.10
 - csv-demo-eslint@0.1.15
 - csv-demo-esm@0.0.24
 - csv-issues-cjs@0.1.11
 - csv-issues-esm@0.0.15
 - csv-demo-ts-moduleresolution-node16-cjs@0.2.10
 - csv-demo-ts-module-node16@0.2.10
 - csv-demo-webpack-ts@0.1.12
 - csv-demo-webpack@0.1.14
 - csv-generate@4.2.7
 - csv-parse@5.4.1
 - csv-stringify@6.4.1
 - csv@6.3.2
 - stream-transform@3.2.7

* feat(csv-issues-cjs): 399 issue

* fix(csv-demo-ts-cjs-node16): upgrade module definition after latest typescript

* feat(csv-parse): new comment_no_infix option (fix adaltas#325)

* test(csv-issues-esm): reproduce issue adaltas#391

* refactor(csv-stringify): rename variable in sample

* test(csv-issues-cjs): reproduce issue 327

* chore(release): publish

 - csv-demo-browser@0.1.13
 - csv-demo-cjs@0.2.11
 - csv-demo-eslint@0.1.16
 - csv-demo-esm@0.0.25
 - csv-issues-cjs@0.2.0
 - csv-issues-esm@0.0.16
 - csv-demo-ts-cjs-node16@0.2.11
 - csv-demo-ts-module-node16@0.2.11
 - csv-demo-webpack-ts@0.1.13
 - csv-demo-webpack@0.1.15
 - csv-generate@4.2.8
 - csv-parse@5.5.0
 - csv-stringify@6.4.2
 - csv@6.3.3
 - stream-transform@3.2.8

* docs(csv-parse): comment_no_infix sample

---------

Co-authored-by: David Worms <david@adaltas.com>
Co-authored-by: Petter <petter@petterhaggholm.net>
Co-authored-by: Mateusz Burzyński <mateuszburzynski@gmail.com>
Co-authored-by: Tom Emelko <tom.emelko@gmail.com>
Co-authored-by: Elia Maino <eliamaino@gmail.com>
Co-authored-by: David Tanner <darthtanner@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants