Issue with pino/thread-stream when packaged

[null-byt3]

What version of pkg are you using?

5.4.1

What version of Node.js are you using?

14.17.6

What operating system are you using?

macOS

What CPU architecture are you using?

Intel Core i7 (2019 macbook pro)

What Node versions, OSs and CPU architectures are you building for?

"linux", "macos", "windows" (default node version above, no other params)

Describe the Bug

There seems to be an issue when packaging and using the transport feature of the pino logging library

events.js:377
      throw er; // Unhandled 'error' event
      ^
Error: Cannot find module '/snapshot/siem/node_modules/thread-stream/lib/worker.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1819:46)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:187:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:399:24)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
Emitted 'error' event on process instance at:
    at emitUnhandledRejectionOrErr (internal/event_target.js:579:11)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:403:9)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []

It seems to originate from pino, a logging library i'm using, but i think the underlying problem comes from the thread-stream library.

I tried manually adding '/node_modules/thread-stream/lib/worker.js' to 'scripts' in package.json, however then this error shows up:

events.js:377
      throw er; // Unhandled 'error' event
      ^

TypeError: Invalid host defined options
    at start (/snapshot/siem/node_modules/thread-stream/lib/worker.js)
    at Object.<anonymous> (/snapshot/siem/node_modules/thread-stream/lib/worker.js)
    at Module._compile (pkg/prelude/bootstrap.js:1794:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at MessagePort.<anonymous> (internal/main/worker_thread.js:187:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:399:24)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
Emitted 'error' event on ThreadStream instance at:
    at /snapshot/siem/node_modules/thread-stream/index.js:155:16
    at processTicksAndRejections (internal/process/task_queues.js:77:11)

When running the project through node it works as expected, no errors.
Also when running the pino logger without the transport feature.

Expected Behavior

Not to throw an error.

To Reproduce

Can be reproduced by packaging a script that uses the pino library, initializes a logger instance, and logs a message.
The error appears when using the logger instance, for example:

const pino = require('pino');
const pinoLogger = pino({
  transport: {
    target: 'pino-pretty',
    options: { translateTime: true, colorize: true },
  },
});


pinoLogger.info('some logging message'); // Throws error

[joshdawson]

I think I'm seeing the same issuee.

The differences for me are:

• I'm using pnpm so my stack-trace is a little different.
• I'm on node 16.
• My CPU architecture is Apple silicon

Everything works fine if I don't initialise pino with a transport.

node:internal/event_target:777
  process.nextTick(() => { throw err; });
                           ^
Error: Cannot find module '/snapshot/integration/node_modules/.pnpm/thread-stream@0.13.0/node_modules/thread-stream/lib/worker.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1819:46)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at MessagePort.<anonymous> (node:internal/main/worker_thread:187:24)
    at MessagePort.[nodejs.internal.kHybridDispatch] (node:internal/event_target:562:20)
    at MessagePort.exports.emitMessage (node:internal/per_context/messageport:23:28)
Emitted 'error' event on Worker instance at:
    at Worker.[kOnErrorMessage] (node:internal/worker:289:10)
    at Worker.[kOnMessage] (node:internal/worker:300:37)
    at MessagePort.<anonymous> (node:internal/worker:201:57)
    at MessagePort.[nodejs.internal.kHybridDispatch] (node:internal/event_target:562:20)
    at MessagePort.exports.emitMessage (node:internal/per_context/messageport:23:28) {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

[robertsLando]

To make pino working add this to package.json:

"pkg": {
    "scripts": [
      "./node_modules/thread-stream/lib/worker.js",
      "./node_modules/pino/lib/worker.js"
    ],
    "patches": {
			"./node_modules/thread-stream/lib/worker.js": ["fn = (await realImport(workerData.filename)).default", "fn = realRequire(workerData.filename.replace('file://', ''))"],
			"./node_modules/pino/lib/worker.js": ["fn = (await realImport(toLoad)).default", "fn = realRequire(t.target)"]
		}
  },

Why?

Pino uses worker threads starting from v7, the problem is that to require worker threads it don't uses require or import but it uses realRequire package, this makes pkg to don't include workers files in the package application. The problem then is that the calls using realImport fail as the path is something like file:///snapshot/path/to/worker.js, note the /// in the initial part, the error then is not cached and so require is never done, this is fixed by pathes that tells pkg to patch those files by find replace those strings (replace the first with the second one). This made me lost a day but it's working now

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[github-actions]

This issue is now closed due to inactivity, you can of course reopen or reference this issue if you see fit.

[jord1e]

For future reference, instead of defining the pino-pretty transport in the application itself:

transport: {
    target: 'pino-pretty',
    options: {
        colorize: true
    }
}

I pipe it into the commandline-utility:

node www.js | yarn pino-pretty

This works, and this weird error is gone

[meppig]

To make pino working add this to package.json:

"pkg": {
    "scripts": [
      "./node_modules/thread-stream/lib/worker.js",
      "./node_modules/pino/lib/worker.js"
    ],
    "patches": {
			"./node_modules/thread-stream/lib/worker.js": ["fn = (await realImport(workerData.filename)).default", "fn = realRequire(workerData.filename.replace('file://', ''))"],
			"./node_modules/pino/lib/worker.js": ["fn = (await realImport(toLoad)).default", "fn = realRequire(t.target)"]
		}
  },

Why?

Pino uses worker threads starting from v7, the problem is that to require worker threads it don't uses require or import but it uses realRequire package, this makes pkg to don't include workers files in the package application. The problem then is that the calls using realImport fail as the path is something like file:///snapshot/path/to/worker.js, note the /// in the initial part, the error then is not cached and so require is never done, this is fixed by pathes that tells pkg to patch those files by find replace those strings (replace the first with the second one). This made me lost a day but it's working now

In case anyone is having that issue after updating to pino 7.8.1, I got pkg working again with the following config:

  "pkg": {
    "scripts": [
      "./node_modules/thread-stream/lib/worker.js"
    ],
    "targets": [
      "node16-win-x64"
    ],
    "patches": {
      "./node_modules/thread-stream/lib/worker.js": [
        "fn = (await realImport(filename))",
        "fn = realRequire(filename.replace(/file:\\/\\/\\/?/, ''))"
      ],
      "./node_modules/pino/lib/transport-stream.js": [
        "fn = (await realImport(toLoad))",
        "fn = realRequire(target)"
      ]
    }
  },

[robertsLando]

@meppig You configuration has an error:

"fn = realRequire(filename.replace(/file:\\/\\/\\/?/, ''))"

should be

"fn = realRequire(filename.replace(/file:\\/\\/?/, ''))"

otherwise it will strip out also leading / from /snapshot/... and require will fail:

node:internal/event_target:646
  process.nextTick(() => { throw err; });
                           ^

Error: Cannot find module 'snapshot/test-pino/node_modules/pino/lib/worker.js'
Require stack:
- /snapshot/test-pino/node_modules/real-require/src/index.js
- /snapshot/test-pino/node_modules/thread-stream/lib/worker.js

[meppig]

@robertsLando Thanks for your follow-up.
I'm building for Windows and i had to replace file:/// URIs with triple slashes, could it be that you are on Linux? I did not really investigate further when it worked since I was happy that I could focus on making progress with my actual project again.
I had to modify your original solution slightly as well to make it work on my machine. I guess that without your post I would still be stuck in nomansland -. with your answer I knew where to look and you made a good example on how to patch these files. So thank you again here :-)

[robertsLando]

could it be that you are on Linux

Yep I'm on Linux! It could be that so because paths are different in such cases. Thanks to you, will try to keep this answer updated as pino is upgraded :)