pkg executable runs differently on 4.4.9 and 4.5.1 #1130

stevenalbert · 2021-04-13T14:12:19Z

Overview minimum project:

Basic Ts.ED framework from cli
Using tsc and babel to make js files
Using pkg to make executable from js transpiled files

Issue:
Create executable on windows with version 4.5.1 gives different result than version 4.4.9.
On version 4.4.9, executables resulted 3 routes (1 explicitly imported and 2 is added from controllers directory).
On version 4.5.1, executables resulted 1 routes (1 explicitly imported only), whereas 2 from controllers directory is not generated.
After looking for a while, it's like controllers not read on version 4.5.1.
I tested on version 4.5.0 too with the result same as 4.5.1

I try to debug the difference, and I added line on node_modules\@tsed\di\lib\utils\importFiles.js and the log can be seen on logs below:

    ...
    const files = globby_1.default.sync(cleanGlobPatterns_1.cleanGlobPatterns(patterns, exclude));
    console.log(patterns, exclude, files)
    ...

Project directory:

package.json:

  "scripts": {
    "build": "npm run tsc",
    "build:babel": "babel ./dist --out-dir dist",
    "build:pkg": "pkg package.json",
    "build:pkg44": "npm install pkg@4.4.9 && npm run build:pkg:all",
    "build:pkg45": "npm install pkg@4.5.1 && npm run build:pkg:all",
    "build:pkg:all": "npm run build && npm run build:babel && npm run build:pkg",
    "tsc": "tsc --project tsconfig.compile.json",
  },
  "pkg": {
    "scripts": "dist/**/*.js",
    "targets": [
      "node14-win-x64"
    ],
    "outputPath": "pkg"
  },

Using version 4.4.9, I can get 3 routes:

PS D:\Data\test-pkg> npm run build:pkg44

> test-pkg@1.0.0 build:pkg44
> npm install pkg@4.4.9 && npm run build:pkg:all


up to date, audited 681 packages in 2s

20 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> test-pkg@1.0.0 build:pkg:all
> npm run build && npm run build:babel && npm run build:pkg


> test-pkg@1.0.0 build
> npm run tsc


> test-pkg@1.0.0 tsc
> tsc --project tsconfig.compile.json


> test-pkg@1.0.0 build:babel
> babel ./dist --out-dir dist

Successfully compiled 4 files with Babel (425ms).

> test-pkg@1.0.0 build:pkg
> pkg package.json

> pkg@4.4.9
PS D:\Data\test-pkg> .\test-pkg.exe
[2021-04-13T20:49:32.457] [DEBUG] [TSED] - Start server...
D:\snapshot\test-pkg\dist/controllers/**/*.ts [ '**/*.spec.ts' ] [
  'D:/snapshot/test-pkg/dist/controllers/HelloWorldController.js',
  'D:/snapshot/test-pkg/dist/controllers/pages/IndexController.js'
]
D:\snapshot\test-pkg\dist/mvc/**/*.ts [ '**/*.spec.ts' ] []
D:\snapshot\test-pkg\dist/services/**/*.ts [ '**/*.spec.ts' ] []
D:\snapshot\test-pkg\dist/middlewares/**/*.ts [ '**/*.spec.ts' ] []
[2021-04-13T20:49:32.483] [INFO ] [TSED] - Call hook $beforeInit
[2021-04-13T20:49:32.483] [INFO ] [TSED] - Build providers
[2021-04-13T20:49:32.508] [INFO ] [TSED] - Call hook $afterInit
[2021-04-13T20:49:32.508] [INFO ] [TSED] - Mount app context
[2021-04-13T20:49:32.509] [INFO ] [TSED] - Mount app router
[2021-04-13T20:49:32.511] [INFO ] [TSED] - Load routes
[2021-04-13T20:49:32.511] [INFO ] [TSED] - Call hook $beforeRoutesInit
[2021-04-13T20:49:32.517] [INFO ] [TSED] - Call hook $onRoutesInit
[2021-04-13T20:49:32.518] [INFO ] [TSED] - Call hook $afterRoutesInit
[2021-04-13T20:49:32.518] [INFO ] [TSED] - Routes mounted :
[2021-04-13T20:49:32.519] [INFO ] [TSED] -
┌───────────────┬────────────────────┬────────────────────────────┐
│ Method        │ Endpoint           │ Class method               │
│───────────────│────────────────────│────────────────────────────│
│ GET           │ /rest/hello-world/ │ HelloWorldController.get() │
│───────────────│────────────────────│────────────────────────────│
│ GET           │ /rest/             │ IndexCtrl.get()            │
│───────────────│────────────────────│────────────────────────────│
│ GET           │ /                  │ IndexCtrl.get()            │
└───────────────┴────────────────────┴────────────────────────────┘
[2021-04-13T20:49:32.519] [INFO ] [TSED] - Call hook $beforeListen
[2021-04-13T20:49:32.528] [INFO ] [TSED] - Listen server on http://0.0.0.0:8083
[2021-04-13T20:49:32.528] [INFO ] [TSED] - Call hook $afterListen
[2021-04-13T20:49:32.529] [INFO ] [TSED] - Call hook $onReady
[2021-04-13T20:49:32.531] [INFO ] [TSED] - [default] Swagger JSON is available on http://0.0.0.0:8083/v2/docs/swagger.json
[2021-04-13T20:49:32.531] [INFO ] [TSED] - [default] Swagger UI is available on http://0.0.0.0:8083/v2/docs/
[2021-04-13T20:49:32.535] [INFO ] [TSED] - Started in 76 ms

Using version 4.5.1, I can only get 1 route which is explicitly imported:

PS D:\Data\test-pkg> npm run build:pkg45

> test-pkg@1.0.0 build:pkg45
> npm install pkg@4.5.1 && npm run build:pkg:all


added 51 packages, changed 1 package, and audited 732 packages in 4s

26 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> test-pkg@1.0.0 build:pkg:all
> npm run build && npm run build:babel && npm run build:pkg


> test-pkg@1.0.0 build
> npm run tsc


> test-pkg@1.0.0 tsc
> tsc --project tsconfig.compile.json


> test-pkg@1.0.0 build:babel
> babel ./dist --out-dir dist

Successfully compiled 4 files with Babel (425ms).

> test-pkg@1.0.0 build:pkg
> pkg package.json

> pkg@4.5.1
PS D:\Data\test-pkg> .\pkg\test-pkg.exe
[2021-04-13T21:02:33.791] [DEBUG] [TSED] - Start server...
D:\snapshot\test-pkg\dist/controllers/**/*.ts [ '**/*.spec.ts' ] []
D:\snapshot\test-pkg\dist/mvc/**/*.ts [ '**/*.spec.ts' ] []
D:\snapshot\test-pkg\dist/services/**/*.ts [ '**/*.spec.ts' ] []
D:\snapshot\test-pkg\dist/middlewares/**/*.ts [ '**/*.spec.ts' ] []
[2021-04-13T21:02:33.819] [INFO ] [TSED] - Call hook $beforeInit
[2021-04-13T21:02:33.820] [INFO ] [TSED] - Build providers
[2021-04-13T21:02:33.846] [INFO ] [TSED] - Call hook $afterInit
[2021-04-13T21:02:33.847] [INFO ] [TSED] - Mount app context
[2021-04-13T21:02:33.847] [INFO ] [TSED] - Mount app router
[2021-04-13T21:02:33.848] [INFO ] [TSED] - Load routes
[2021-04-13T21:02:33.851] [INFO ] [TSED] - Call hook $beforeRoutesInit
[2021-04-13T21:02:33.853] [INFO ] [TSED] - Call hook $onRoutesInit
[2021-04-13T21:02:33.854] [INFO ] [TSED] - Call hook $afterRoutesInit
[2021-04-13T21:02:33.854] [INFO ] [TSED] - Routes mounted :
[2021-04-13T21:02:33.855] [INFO ] [TSED] -
┌───────────────┬──────────┬─────────────────┐
│ Method        │ Endpoint │ Class method    │
│───────────────│──────────│─────────────────│
│ GET           │ /        │ IndexCtrl.get() │
└───────────────┴──────────┴─────────────────┘
[2021-04-13T21:02:33.855] [INFO ] [TSED] - Call hook $beforeListen
[2021-04-13T21:02:33.864] [INFO ] [TSED] - Listen server on http://0.0.0.0:8083
[2021-04-13T21:02:33.864] [INFO ] [TSED] - Call hook $afterListen
[2021-04-13T21:02:33.865] [INFO ] [TSED] - Call hook $onReady
[2021-04-13T21:02:33.872] [INFO ] [TSED] - [default] Swagger JSON is available on http://0.0.0.0:8083/v2/docs/swagger.json
[2021-04-13T21:02:33.872] [INFO ] [TSED] - [default] Swagger UI is available on http://0.0.0.0:8083/v2/docs/
[2021-04-13T21:02:33.874] [INFO ] [TSED] - Started in 80 ms

The text was updated successfully, but these errors were encountered:

robertsLando · 2021-04-13T16:06:52Z

@erossignon any clue?

erossignon · 2021-04-14T13:48:49Z

@stevenalbert, could you try running with /debug option on so we could compare the two logs.

stevenalbert · 2021-04-15T14:58:43Z

@erossignon
Since the debug is long, I attached them here.

Version 4.4.9:
pkg44-debug.txt

Version 4.5.1:
pkg45-debug.txt

erossignon · 2021-04-17T07:09:44Z

@stevenalbert , thank you, the debug output are identical .. My guess is that the issue is not in the walker , but in the bootstrap.

I also spotted that the file starts with "D:" on both examples.

Do you have the same issue if you run on the C:\ drive instead ?

stevenalbert · 2021-04-17T14:10:25Z

@erossignon Yes, I just tried to build and run on C:\ drive, the issue still exists.

dominics · 2021-05-04T12:02:13Z

Hello! I've put together a simple reproduction of the problem at https://github.com/dominics/bug-pkg-globby - it's a small three-line script, but submitting a repo lets you build it, play around with it, and I can happily provide binaries if anyone can't build it. The reproduction script is just:

const globby = require('globby')
const results = globby.sync(['**/*.js'], { cwd: '/snapshot' })
console.log('Contents of /snapshot: ', results)

This prints hundreds of matches when built under 4.4.9, and zero matches when built under 4.5.0 (or 5.1.0 for that matter), even when the snapshot contents are identical. From this reproduction repo, I've learned:

globby can't match anything in /snapshot - this is why e.g. oclif is not picking up people's commands when they're packaged in the snapshot (for example, support pkg little better with default setup oclif/oclif#177 (comment))
the problem was introduced in 4.5.0 (before 4.5.1 - can change the title of this issue)
it affects macOS, Alpine and Linux targets too
@erossignon you're correct, definitely not the walker, and [feat] Replace internal walker with nft #1138 would not fix this - probably an issue with fs.readdirSync/fs.readdir patching

Note: trying to use 4.4.9 seems to run into the pkg-fetch related recommendation to upgrade to 5.x, which is pretty annoying when combined with this regression! The reproduction repo uses yarn resolutions as a workaround.

github-actions · 2021-08-03T00:11:55Z

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

dominics · 2021-08-03T11:26:13Z

This is still valid and relevant, bot.

tdankle · 2021-08-11T22:46:07Z

FWIW, I upgraded pkg from 4.4.9 to 5.3.1 and unfortunately, my app.exe no longer runs. Prior to finding this open issue, I traced it down to the fact that my code crawls through a directory within the snapshot: i.e.: via:

fs.readdirsync ( path.join(__dirname, './handlers') )

This call fails immediately with the message: "ENOENT, no such file or directory, scandir..."

(The message includes the exact directory that is included in the snapshot -- I've confirmed it via --debug).

So, it's definitely a problem with the patching of readdirsync, reader as @dominics pointed out above.

tdankle · 2021-08-12T05:08:56Z

My pkg compiled app had been working 'swimmingly' for years. As described above, I upgraded from 4.4.9 to latest and it breaks on a readdirSync.

Unfortunately, if you perform this upgrade, you cannot REVERT back primarily because you'll end up deleting your cached pkg-fetch based binaries and when you `npm install pkg@4.4.9', you'll quickly discover why your screwed: For this older version, it's pkg-fetch attempts to retrieve from an old 'zest' location that no longer exists. Ugh.!!!

In any event, being forced to figure out a way forward, the described issue involves the following:

Within pkg's bootstrap.js it hijacks node's fs.readdirsync with a replacement that handles the fs.readdirsync option named 'withFileTypes' incorrectly. In particular, the statement (currently v5.3.1) line 1148:

if (!options || options.withFileTypes) {

should be changed to:

if (!options) {

And... That's the fix.

roberttod · 2021-08-26T16:36:34Z

Thanks @tdankle , I had this issue and that works like a charm. I opened a PR #1299

github-actions · 2021-11-25T00:13:33Z

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

dominics · 2021-11-25T01:06:50Z

Commenting to avoid close. Not stale, PR open.

github-actions · 2022-02-24T00:17:49Z

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

github-actions · 2022-03-01T00:19:50Z

This issue is now closed due to inactivity, you can of course reopen or reference this issue if you see fit.

dominics · 2022-08-25T00:38:10Z

Thank you so much @kldzj and @jesec! Fantastic work

robertsLando mentioned this issue Apr 16, 2021

[feat] Replace internal walker with nft #1138

Closed

dominics mentioned this issue May 4, 2021

support pkg little better with default setup oclif/oclif#177

Closed

github-actions bot added the Stale label Aug 3, 2021

robertsLando removed the Stale label Aug 3, 2021

roberttod mentioned this issue Aug 26, 2021

Fix fs readdir and readdirSync #1299

Closed

github-actions bot added the Stale label Nov 25, 2021

github-actions bot removed the Stale label Nov 26, 2021

kldzj mentioned this issue Jan 30, 2022

fix: fs readdir and readdirSync (#1130/#1299) #1495

Merged

jesec pushed a commit that referenced this issue Feb 1, 2022

fix: bootstrap: fs readdir and readdirSync (#1130/#1299) (#1495)

5010411

github-actions bot added the Stale label Feb 24, 2022

github-actions bot closed this as completed Mar 1, 2022

gjermundgaraba mentioned this issue Mar 13, 2022

bring ARM support ignite/cli#2130

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg executable runs differently on 4.4.9 and 4.5.1 #1130

pkg executable runs differently on 4.4.9 and 4.5.1 #1130

stevenalbert commented Apr 13, 2021 •

edited

Loading

robertsLando commented Apr 13, 2021

erossignon commented Apr 14, 2021

stevenalbert commented Apr 15, 2021

erossignon commented Apr 17, 2021

stevenalbert commented Apr 17, 2021

dominics commented May 4, 2021 •

edited

Loading

github-actions bot commented Aug 3, 2021

dominics commented Aug 3, 2021

tdankle commented Aug 11, 2021

tdankle commented Aug 12, 2021

roberttod commented Aug 26, 2021

github-actions bot commented Nov 25, 2021

dominics commented Nov 25, 2021 •

edited

Loading

github-actions bot commented Feb 24, 2022

github-actions bot commented Mar 1, 2022

dominics commented Aug 25, 2022

pkg executable runs differently on 4.4.9 and 4.5.1 #1130

pkg executable runs differently on 4.4.9 and 4.5.1 #1130

Comments

stevenalbert commented Apr 13, 2021 • edited Loading

robertsLando commented Apr 13, 2021

erossignon commented Apr 14, 2021

stevenalbert commented Apr 15, 2021

erossignon commented Apr 17, 2021

stevenalbert commented Apr 17, 2021

dominics commented May 4, 2021 • edited Loading

github-actions bot commented Aug 3, 2021

dominics commented Aug 3, 2021

tdankle commented Aug 11, 2021

tdankle commented Aug 12, 2021

roberttod commented Aug 26, 2021

github-actions bot commented Nov 25, 2021

dominics commented Nov 25, 2021 • edited Loading

github-actions bot commented Feb 24, 2022

github-actions bot commented Mar 1, 2022

dominics commented Aug 25, 2022

stevenalbert commented Apr 13, 2021 •

edited

Loading

dominics commented May 4, 2021 •

edited

Loading

dominics commented Nov 25, 2021 •

edited

Loading