Skip to content

Commit

Permalink
[FIX] Ensure proper handling of multi-byte characters in streams
Browse files Browse the repository at this point in the history
The stream encoding needs to be set in order to properly handle
multi-byte utf8 characters that are split between different chunks.

See:
https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding
  • Loading branch information
matz3 committed Feb 4, 2020
1 parent 7778160 commit d84b5b2
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions lib/processors/stringReplacer.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@ const replaceStream = require("replacestream");
*/
module.exports = function({resources, options}) {
return Promise.all(resources.map((resource) => {
const stream = resource.getStream()
.pipe(replaceStream(options.pattern, options.replacement));
const stream = resource.getStream();
stream.setEncoding("utf8");
stream.pipe(replaceStream(options.pattern, options.replacement));

resource.setStream(stream);
return resource;
Expand Down

0 comments on commit d84b5b2

Please sign in to comment.