Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎯 ESM Support for PnP #638

Closed
zdm opened this issue Dec 17, 2019 · 34 comments · Fixed by #2161
Closed

🎯 ESM Support for PnP #638

zdm opened this issue Dec 17, 2019 · 34 comments · Fixed by #2161
Assignees
Labels
case study Package compatibility report esm

Comments

@zdm
Copy link

zdm commented Dec 17, 2019

Edit 2021-04-14 by @arcanis: In case you reach this thread by Twitter, ESM for PnP is on our roadmap. It's tricky due to some primitives currently missing in Node (they're working on this, but as you can see it's still branded as experimental), in the meantime if you need it then the node-modules linker remain a perfectly fine solution. In any case, help is appreciated, Twitter rants less.


What package is covered by this investigations?```
@softvisio/cli

Describe the goal of the investigation

I am trying to run script, that uses es6 syntax.

yarn add @softvisio/cli
yarn softvisio-cli

But get this error:

(node:4268) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
d:\downloads\yarn\@softvisio-cli-npm-0.1.0-efe3491d08-1.zip\node_modules\@softvisio\cli\bin\cli.js:3
import softvisioCliApi from "../lib/api.js";
^^^^^^

I am using node 13 that can load es6 modules directly.

Is it currently possible to use pure es6 modules with yarn 2 or I need to compile them to commonjs before?

@zdm zdm added the case study Package compatibility report label Dec 17, 2019
@arcanis
Copy link
Member

arcanis commented Dec 17, 2019

ESM modules go through a completely different codepath in Node (cjs / esm). The problem is that this codepath is still very new, and although modules have technically been marked stable the way to alter the resolution (aka loaders) is still in heavy discussion (nodejs/modules#351).

So we can't support them at the moment (too early), but as soon as the loader spec moves forward with an implementation we'll be ready to experiment with them 🧪

@zdm
Copy link
Author

zdm commented Dec 17, 2019

Thank you.
Please, don't close this issue until this feature will be implemented.

@arcanis arcanis changed the title [Case Study] How to load pure es6 module [Case Study] ESM Support Apr 22, 2020
@arcanis arcanis added the help wanted Extra attention is needed label Apr 22, 2020
@arcanis arcanis pinned this issue Apr 22, 2020
@arcanis arcanis changed the title [Case Study] ESM Support 🎯 ESM Support Apr 22, 2020
@bgotink
Copy link
Member

bgotink commented Apr 23, 2020

I looked into the API a bit using Node 14. I've written a PoC implementation at (bgotink@34c0a7b) along with a small test repo over at https://github.com/bgotink/berry-esm.

The API is still unstable, but I think it's worth checking out to see whether there's any valuable feedback to gain from trying. I'm not sure how volatile we should consider the API, so I'm not considering "should we ship this?" at the moment, only "does it work?"

The interesting bits so far:

  • It works.
  • Resolving a CommonJS dependency using {format: 'commonjs'} in the getFormat hook gives us no more hooks into the file's resolution, so we can't load it from the yarn cache's zipfiles. The same goes for JSON files and {format: 'json'}. All of these need to go through {format: 'dynamic'} which is kinda sad because that means we need to replicate the Node behaviour w.r.t. named exports in these kinds of files.
  • This will go nicely with support for package exports ([Case Study] Exports Support #650), where the regular .pnp.c?js file asks for CJS files and the ESM loader uses import. In fact, I would assume most packages that want to adopt ESM will do this in a backwards compatible way, in which case we need package exports to load the ESM variant of the library.

The boring bits:

  • Passing the --experimental-loader flag logs a warning regardless of whether said loader is actually ever used. In other words, so to not harass users not interested in experimental ESM support we should make this opt-in
  • This is probably not compatible with the native ESM loader. For instance, the loader powered by PnP is quite lenient w.r.t. file extensions, while the native ESM loader isn't.
  • Some things don't work yet, e.g. import(await import.meta.resolve('typescript')) fails at the moment because the ?yarn-cache flag gets lost in the resolver
  • Stuff like import(pathToFileURL(createRequire(import.meta.url).resolve('typescript'))) doesn't work.
  • I've explicitly kept the pnp-loader static and use the PnP API exposed from the .pnp.c?js file

Improvements to the PnP API that would make the PoC a lot cleaner:

  • Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)
  • It would be great if pnp could be given an option to make file extensions required when directly referencing a file
  • A function to see whether a given file path is part of the yarn cache or not would come in handy, that would allow us to use regular file: URLs instead of using a ?yarn-cache query parameter.

A node module with code below works fine in the berry root folder and showcases a lot of the basic functionality already.

// named exports from fs still work even though we patch fs
import fs, {readFileSync} from 'fs';

// JSON can be imported
import pkg                from 'typescript/package';
// CJS package import
import typescript         from 'typescript';

// importing a non-patched builtin
import {fileURLToPath}    from 'url';

console.log('');
console.log(pkg.name, typescript.version);

// Resolves to a regular file: URL
const typescriptUrl = await import.meta.resolve('typescript');
// So it can get transformed into a path and used in fs APIs or even require (BAD IDEA)
const typescriptPath = fileURLToPath(typescriptUrl);

console.log(`url:`, typescriptUrl);
console.log(`path:`, typescriptPath);

const eslintPackagePath = fileURLToPath(await import.meta.resolve('eslint/package'));

// Just to prove that the fs APIs are patched
console.log('fs.readFileSync', JSON.parse(fs.readFileSync(eslintPackagePath)).version);
console.log('readFileSync', JSON.parse(readFileSync(eslintPackagePath)).version);
$ node --experimental-loader ./.pnp-loader.mjs --experimental-import-meta-resolve --harmony-top-level-await test.mjs
(node:68602) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

typescript 3.8.3
url: file:///Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js?yarn-cache=
path: /Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js
fs.readFileSync 6.8.0
readFileSync 6.8.0

@arcanis
Copy link
Member

arcanis commented Apr 27, 2020

Nice analysis! You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)

That seems acceptable 👍

It would be great if pnp could be given an option to make file extensions required when directly referencing a file

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

@bgotink
Copy link
Member

bgotink commented Apr 27, 2020

You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

I've kept the module loader conservative in which paths it actually tries to handle itself versus which paths it lets the default loader handle. This requires the different hooks to be able to check whether a URL points to a path that needs to be loaded via the patched filesystem.
The ?yarn-cache query param was a quick and dirty solution for that. Another possibility is an API to see whether a given path requires loading via our patched fs module. Alternatively we could just load all files via the patched fs, though that makes the loader anything but conservative.
In a first attempt for the loader I used a custom protocol yarn-cache:, but it's a bad idea to stray away from the file: protocol for compatibility reasons: import.meta.resolve(...) and import.meta.url expose the URL of a module, changing that from a file: URL to something else will cause breakage for people using the module's URL in order to e.g. load a data file in the same folder as the code file.

I'm not sure what's the best path forward here: just load everything via our getSource hook or defer to the native hook when possible? (in the latter case we need a mechanism to devise whether a URL should be loaded by us, beit via the current ?yarn-cache query parameter or via an API call)

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

resolveToUnqualified doesn't resolve bare imports to their actual files, it resolves typescript to /.../berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript.
At the moment the PoC loader uses resolveQualified to resolve that to an actual file, but that implements too much: it automatically adds file extensions and it resolves folders to files called index.whatever.

As I've stated in my previous comment I've kept the module loader as dumb as possible, it loads everything from the regular .pnp.c?js file. Implementing both the CJS resolution algorithm and the ESM resolution algorithm in the regular pnp file is probably not the best idea. Maybe the solution is to use resolveToUnqualified and then handle the rest of the resolution in the module loader?

@dezren39
Copy link

What is the timeline for enabling ES support? Is there anything one can do to work around this? The issues linked to this suggest not. For now, I will try using berry with the nodeLinker option and see if that lets me use berry, though with it's biggest feature limited lol. The reason I want pnp is essentially what you've labeled 'zero installs', before this I used yarn 1 and the offline mirror mode to make exactly the same .zip folders that berry makes, even in the .yarn folder and everything lol allowing 1 zip per dependency and sane commit changes by just checking the .zips. Built it all with docker images lol. So I'd hoped berry would be able to work in the same way, but since there actually isn't ever a node_modules with this method, I see how it will be more difficult to patch, because the offline-mirror was just how it got stored in git, but once running it was identical to npm/node with a node_modules folder.

@desmap
Copy link

desmap commented Jul 23, 2020

@arcanis @bgotink any news here? @arcanis left the last comment on the related thread nodejs/modules#351 on on Jul 29, 2019, so is there some hope?

@arcanis
Copy link
Member

arcanis commented Jul 23, 2020

Of course there's hope, and it's even certain it'll eventually happen. It's however not my top priority at the moment since all the software I personally use are still CommonJS, so it'll happen faster if someone can do the legwork 🙂

@desmap
Copy link

desmap commented Jul 25, 2020

I personally use are still CommonJS

Most of the stuff I use is also CommonJS, I agree. However, once you start to code share between frontend and backend and already bigger libs like Next default to esnext modules it's just cleaner and less headache to have all the code base as one format.

Currently, code sharing is a rocket science. But that's not your fault ;)

@desmap
Copy link

desmap commented Jul 25, 2020

or another example which just popped up again in my project: top-level awaits, they are such a life saver and way underrated. Once you get the hang on them you can do insane stuff. Right now I try to setup conditional imports (which are of course async) and then continue depending on the condition. With top-level it would be DRY, without, a lot of .thens, code dupes and just messy.

But TS requires esnext modules for using top-level await... so yeah

@nnmrts

This comment has been minimized.

@arcanis
Copy link
Member

arcanis commented Nov 8, 2020

Hey @nnmrts - before commenting in an open source project (any open source project), please remember that the people in front of you aren't your contractors but your peers. Passive aggressiveness like "if you don't do X I won't use Y" isn't more likely to make X happen, because maintainers don't work for you (at least I haven't received the wire yet).

That being said, if you need to use ESM now, you just have to downgrade to typical node_modules installs using the nodeLinker setting and you'll have your work done for you. If you need to use ESM with PnP then it's perfect as we happen to be looking for someone with free time to help us on this. Ping us on Discord and we'll put your energy to good use for the community! 🙂

@merceyz
Copy link
Member

merceyz commented Nov 22, 2020

I gave this a go and implemented an experimental loader in #2161, if anyone would like to test/try it you can use yarn set version from sources --branch 2161

@zdm zdm closed this as completed Sep 8, 2021
@merceyz merceyz reopened this Sep 8, 2021
lifehome added a commit to lifehome/whmcs_stockchecker that referenced this issue Sep 11, 2021
As of commit time, yarn2 cannot support ESM scripts due to difficulties from upstream(node itself).
See: yarnpkg/berry#638
lifehome added a commit to lifehome/whmcs_stockchecker that referenced this issue Sep 11, 2021
As of commit time, yarn2 cannot support ESM scripts due to difficulties from upstream(node itself).
See: yarnpkg/berry#638
UberOpenSourceBot pushed a commit to fusionjs/fusionjs that referenced this issue Sep 16, 2021
Adding support for pure ESM packages ([see](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c)). This is achieved by using proper external type (`commonjs` vs. `module`) in webpack's external config, so that proper code is generated to import ES modules in the server bundle. A few important facts:
* Node.js requirement had to be bumped to >= 12.17, in order to use the dynamic `import()`
* The `server-main.js` cjs entry becomes an "async module", so it returns a Promise that need to be awaited when using `require()`
* Yarn PnP does not support ESM yet (see yarnpkg/berry#638); we'll bundle esm dependencies on the server to workaround this when project is using PnP (the `process.versions.pnp` check)

Co-authored-by: Ryan Tsao <ryan.j.tsao@gmail.com>
minimabot added a commit to minimabot/mocheong_deprecated that referenced this issue Oct 18, 2021
minimabot added a commit to minimabot/mocheong_deprecated that referenced this issue Oct 25, 2021
@merceyz merceyz added the esm label Oct 28, 2021
@merceyz merceyz unpinned this issue Oct 28, 2021
@merceyz merceyz removed the help wanted Extra attention is needed label Oct 28, 2021
@eric-burel
Copy link

eric-burel commented May 11, 2022

Hello from 2022, I feel at tad lost among all tickets and doc, am I supposed to do something to have for instance an index.mjs files that imports zx/globals? I hit Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'zx' for instance.
At the moment I fallback to nodeLinker: node_modules and it works ok at least.

@merceyz
Copy link
Member

merceyz commented May 11, 2022

If there are no indicators in your project other than file extensions that ESM support is required you'll need to manually enable it.

pnpEnableEsmLoader: true

https://yarnpkg.com/configuration/yarnrc#pnpEnableEsmLoader

@merceyz merceyz reopened this May 11, 2022
@merceyz merceyz closed this as completed May 11, 2022
@eric-burel
Copy link

Adding type: "module" in package.json also generate an ESM loader.
It seems to be documented only in this blog post atm: https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-packages--3hak#esm-support

However, maybe a dumb question: how do you actually use this loader? I now have a .pnpn.loader.mjs but zx package is still not found when I run node index.mjs. I guess I should also register this loader somewhere? Am I supposed to use manually the functions it exports?

@kristian
Copy link

kristian commented May 12, 2022

node -r .pnp.js --loader .pnpn.loader.mjs index.mjs

Or just use:

yarn node index.mjs

Which does add all required parameters for you.

@eric-burel
Copy link

eric-burel commented May 12, 2022

Perfect thank you!
If I should sum it up for future googlers:

  • If you are migrating from nodeLinker: node-modules, and still have a "node_modules" folder, you might want to add nodeLinker: pnp explicitely so Yarn does the switch (otherwise if you just remove the nodeLinker option, it might still think you are using node modules).
  • Yarn will detect type: "module" in your package.json. If you don't want to add this property, instead in .yarnrc.yml add pnpEnableEsmLoader: true
  • It generates a loader, but you must still use it explictely, with either node -r .pnp.js --loader .pnp.loader.mjs index.mjs (most portable) or yarn node index.mjs
  • ESM is still experimental, for instance I hit this issue [Feature] Support the imports field in the PnP API #3843 with "chalk" which is also used by "zx". But at least I could setup PNP

@nerdgore
Copy link

Thanks for the write up @eric-burel!
This is a great workaround for code dependencies.
Is it possible to somehow use this loader for scripts? In my case I am trying to run plop through a package script

@Samlv9
Copy link

Samlv9 commented May 31, 2022

Not works when i am trying to use a custom loader.

/// file: custom.loader.ts
import { extname } from "node:path";
import { create, createEsmHooks, NodeLoaderHooksAPI2, NodeLoaderHooksFormat } from "ts-node";

interface NodeLoaderHooksAPI2Context {
    conditions?: NodeLoaderHooksAPI2.NodeImportConditions;
    importAssertions?: NodeLoaderHooksAPI2.NodeImportAssertions;
    parentURL: string;
}

interface NodeLoaderHooksAPI2ReturnType {
    url: string;
    format?: NodeLoaderHooksFormat;
    shortCircuit?: boolean;
}

const { resolve: tsnodeResolve } = createEsmHooks(create());

export function resolve(specifier: string, context: NodeLoaderHooksAPI2Context, defaultResolve: NodeLoaderHooksAPI2.ResolveHook): Promise<NodeLoaderHooksAPI2ReturnType> {
    switch (extname(specifier)) {
        case ".ts" :
        case ".tsx" :
            return tsnodeResolve(specifier, context, defaultResolve)

        default: break;
    }

    return defaultResolve(specifier, context, defaultResolve);
}
$ yarn tsc --target esnext --moduleResolution node ./custom.loader.ts
$ yarn node --loader ./custom.loader.js --no-warnings --experimental-json-modules --experimental-import-meta-resolve ./test.ts

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ts-node' imported from xxxxxx
Did you mean to import ts-node-virtual-bba6bca08f/0/cache/ts-node-npm-10.8.0-e60a0a9a4f-1c22dc8dd8.zip/node_modules/ts-node/dist/index.js?
    at new NodeError (internal/errors.js:322:7)
    at packageResolve (internal/modules/esm/resolve.js:732:9)
    at moduleResolve (internal/modules/esm/resolve.js:773:18)
    at Loader.defaultResolve [as _resolve] (internal/modules/esm/resolve.js:887:11)
    at Loader.resolve (internal/modules/esm/loader.js:89:40)
    at Loader.getModuleJob (internal/modules/esm/loader.js:242:28)
    at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:76:40)
    at link (internal/modules/esm/module_job.js:75:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

@merceyz
Copy link
Member

merceyz commented May 31, 2022

@Samlv9 See #4044

@yarnpkg yarnpkg locked as resolved and limited conversation to collaborators May 31, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
case study Package compatibility report esm
Projects
None yet
Development

Successfully merging a pull request may close this issue.