[Case Study] Node 12 Experimental Loader hooks and Yarn P'n'P

[mohsen1]

I couldn't find any resources for how Yarn P'n'P will work with Node 12 Experimental Loader hooks. With that all of the hacks to fs module to make pnp work would be unnecessary. Has design for this started yet?

[arcanis]

Not yet, but it's still on my radar. It might be a post-v2 line, though.

The resolution itself should be fairly easy - the more complex part will be zip loading: we need some way to tell Node that the vendors must be loaded through our hook (rather than through the regular fs as it would usually do). I'm not sure the experimental hooks expose this capability yet.

[rkistner]

It doesn't seem to be supported in Node 12, but Node 13 has a getSource hook, which should address this: https://nodejs.org/docs/latest-v13.x/api/esm.html#esm_code_getsource_code_hook

There's an example to load the source via HTTP, so using the same hook to load from a zip should be easy.

[arcanis]

@rkistner although technically true, it's a bit more complex in that various packages rely on doing things such as calling readFile on their own files (for example to access data files, or sometimes even JSON without going through require).

Ideally Node should expose a package interface that allows packages to access their own sources, but since the very concept of "package" doesn't exist in Node (everything is just modules), it's more difficult. Maybe we could provide the abstraction ourselves, though (similar to #918) ...

[rkistner]

What I would actually find very useful, is a linker mode between node-modules and pnp: Extract all the files to the filesystem, but without the node_modules tree structure - basically pnp mode but unplugging all modules.

It should be possible to implement this using just Node 12's loader hooks, without patching any fs or other modules. Integration with tools such as IDEs should have a lot less issues (e.g. navigate to definition). You'd also get most of the benefits of the pnp mode (no hoisting issues, etc), except for the reduced storage requirements.

[aduh95]

Worth noting that using loader hooks would probably help fixing #906.

[rkistner]

I've started on a proof-of-concept of using loader hooks, coupled with a variation of the pnp installer that unplugs all modules.

It appears to be feasible conceptually, but a major limitation is that Node's loader hooks are currently only supported for ES modules, not for CommonJS modules. This makes it unsuable for most of the current Node ecosystem.

[aduh95]

a major limitation is that Node's loader hooks are currently only supported for ES modules, not for CommonJS modules.

I don't think that's accurate, using getFormat hook, you can support all kind of modules Node.js usually supports according to the docs.

The lack of stability of the feature at the moment, and the fact that it hasn't been fully backported to v10 at the moment make it a hard sell for shipping on v2 imho.

[rkistner]

I don't think that's accurate, using getFormat hook, you can support all kind of modules Node.js usually supports according to the docs.

The getFormat allows you to load a CommonJS module, but any further dependencies of that module will be loaded using the built-in resolver, not the custom loader. I think one of the things making loader hooks complicated for CommonJS modules is that require is a synchronous function, while the loader hooks can be async.

There are lots of discussions on the loader hooks in this thread: nodejs/modules#351

[merceyz]

I'll close this as the loader hooks don't allow us to hook into the fs functions, it does work for resolving and reading modules which is implemented in #2161 but that isn't enough.