Flags functionality options

[GeoffreyBooth]

I thought it might be useful to have a post laying out in neutral terms how the various ESM flag proposals work and the implications of each, to help in discussion for choosing the best approach. If anything in this initial post is incorrect or incomplete or biased, I will edit as appropriate.

There are three options discussed so far for how what was previously called --type should behave. I’ll refer to the options by proposed new names based on how the flag would behave. They all take either module or commonjs as the single accepted and required argument.

1. --input-type would set the type of --eval, --print and STDIN input—and that’s it. If the initial entry point is a file, an error is thrown.

2. --entry-type would set the type of the initial entry point to the program, whether that be a file or one of the non-file input types (--eval etc.); but setting of the type of that initial entry point implies nothing about any other files.

3. --package-type would set or override the "type" field of the package.json file that controls the parsing of the initial entry point (or of the virtual package.json at the root of the volume, if there are no package.json files up the path from the entry point). It would also apply to the non-file input types. Like the "type" field, it would not override any package scopes beyond the one containing the initial entry point.

This isn’t meant to be a discussion of names. I don’t feel that it’s a good use of GitHub issue threads to bikeshed what we should name things. Once we decide on which functionality we want, we can separately determine the best name for it. Please consider the names below to be placeholders.

Pros and cons of each option

--input-type

Pros:

• Provides a way to use ESM in --eval, --print and STDIN, where otherwise ESM syntax would be impossible to enable.

Cons:

• Users would probably expect this to apply to files as well. It’s not obvious why it wouldn’t.

--entry-type

Pros:

• Provides a way to use ESM in “loose” .js or extensionless files, that live outside of any project/package and don’t have a parent package.json.

• Explicitly applies to the file or string being referenced in the node command, so is straightforward in that regard.

Cons:

• So far, all users who have read about this have assumed that setting the type of the entry also opts in to that type for all files imported by that entry point. As in, if you set the entry point to be ESM, import statements of .js files should treat those .js files also as ESM. This user expectation is likely to only become stronger as ESM in browsers becomes more widely used, as this is how <script type="module"> behaves in browsers.

• It is inconsistent for entry.js and dep.js to be side by side, where entry.js imports dep.js, and node --entry-type=module entry.js loads entry.js as ESM but then dep.js is treated as CommonJS. This is the only case where files with the same extension in the same folder are treated differently by Node.

• There’s no use case for overriding the initial entry point of a project while relying on file extensions or package.json to define the type of all other files in the project.

--package-type

Pros:

• Behaves as users expect the flag to behave, by setting the type for an entire project.

• By referencing package.json "type" in its name, this should be easier for users to understand as they should grasp that it behaves the same way as package.json "type" does.

• Allows “loose” .js or extensionless files to import other ESM .js files, so “shell script” .js files don’t need to be limited to a single file.

• Without this, we can’t have --package-type=auto, as it wouldn’t make sense to have type detection for an entry point only. The use case for auto is a project that lacks an explicit "type" field (and uses .js), and it’s implausible to imagine a project with a .js entry point where all other files are .mjs (or in a subfolder under a package.json with a "type" field).

Cons:

• Applies to more than just the file or string passed to node, so users would need to be aware that it’s the equivalent to package.json "type".

Both:

• The only use case for needing this flag for a project is when a project is already using ESM in .js files but without "type": "module"; but most such projects expect Babel or the like to transpile them, and may not be compatible with Node without changes. (For example, they may require refactoring to enable explicit extensions; though --es-module-specifier-resolution=node might be sufficient for most such projects to run without changes.) If we build --package-type=auto, regardless of its effectiveness for Babel ESM projects auto would work great for a CommonJS project without a package.json "type" field.

• Allows opting into ESM mode by default system-wide via NODE_OPTIONS=--package-type=module. After setting such an option in a user’s environment, CommonJS projects would need to either have a "type": "commonjs" in their package.json or be run via --package-type=commonjs. Allowing changing Node to be ESM by default system-wide would be seen as a pro by some and as a con by others; it’s a pro for those who want to leave CommonJS behind and don’t plan on adopting .mjs; and as a con for those who don’t want to encourage people to expect ESM by default and publish projects and packages that assume so. (The latter concern could presumably be addressed somewhat by npm publish checking for import/export syntax in packages about to be published, and erroring if "type": "module" is not present.)

Other considerations

It is not clear to me how these flags would interact with loaders, test runners, stubs/mocks and the like. I’m not sure if any flags are better or worse than others with regard to such things. On the one hand --input-type or --entry-type would seem to be simpler for such add-ons to handle, as they only apply to one string or file; yet if the add-ons need to know how to handle package.json "type", it might be simpler for them to support that and --package-type (which should behave identically) rather than needing to special-case the entry point.

See also

• upstream objection to --type #296 - Initial discussion of upstream objection

• esm: scoped --type, cpp refactoring ecmascript-modules#57 - PR that implements --package-type functionality (though with --type name)

• new ESM implementation node#26745 - Upstream PR, the code for which contains --entry-type currently; and the thread contains some discussion of --type

[jkrems]

I think there's two different cases / functionalities that may be good to keep apart:

1. How should .js be interpreted? Is it .cjs or .mjs?
2. What is the content-type of some bytes that we have no further info on (no extension, stdin, eval, etc.)?

The first is a mostly binary choice between two modes. The latter is not. The content-type could be JSON, WASM, binary module, binary AST (forward looking), JS module, CJS module, or any future format. The first is "switch mime DB", the latter is "provide mime for this specific bit".

If I say node --input-type=wasm <foo.wasm and foo.wasm imports x.js file, we would not want to assume that x.js is WASM just because that was the input type. But if I run node --input-type=wasm --package-mode=js-is-esm <foo.wasm, I would expect that the x.js file imported from foo.wasm will be interpreted as a JS module.

It is not clear to me how these flags would interact with loaders, test runners, stubs/mocks and the like. I’m not sure if any flags are better or worse than others with regard to such things.

For test runners (or coffee --package-type foo.coffee) the problem is that they need to fork to apply this setting to the node runtime, unless it can be changed from inside the process somehow. But we could assume that those tools will just say "If you want to run tests, add a package.json. We don't support --package-type".

[GeoffreyBooth]

Taking off my neutrality hat from the top post, the one thing I’m sure of is that I think --entry-type is a footgun. Most, if not all, users expect it to behave differently than it actually does. Even within our group we have trouble keeping track of how it should behave in various scenarios (no package.json, a package.json that lacks a "type" field, etc.).

I’m sympathetic to the argument that the use cases for --package-type aren’t compelling. Those use cases, that I can think of, are:

• Running a project from a pre-Node 12 that uses import/export syntax and expects Babel or esm, and the user wants to try running it without Babel or esm (and hopefully they’re not doing things that we don’t support, like named exports from CommonJS). There are likely few such projects that will run successfully, so this use case is perhaps not worth considering. If they have to refactor at all, they can add "type": "module".

• Users who want to flip the defaults for Node system-wide to be ESM-first, via NODE_OPTIONS=--package-type=module. To be honest I don’t see why we’d want to prevent users from doing this, but changing Node’s system default has never been a goal of ours, so it’s not necessarily a use case we need to support (yet).

So basically if the choice was only between --entry-type and --input-type, I’d rather ship --input-type. The latter does handle a use case we’ve decided we want to support, namely ESM in --eval/--print/STDIN. That’s important enough to add a flag for, I think we all agree. And --input-type isn’t the footgun that --entry-type is; it simply won’t work on files, rather than operate on files in surprising ways.

Switching from --entry-type to --input-type means that the use case of “loose” .js or extensionless files outside of any package scope would not be executable as is; they would need to be renamed to use .mjs or put inside a folder with a package.json. I’m okay with this. There’s not much JavaScript meant to be run by Node that doesn’t use a non-core dependency, so a file without a package.json should be rare; basically shell scripts or CLI tools, and the latter are usually symlinks into package scopes (like how npm is a symlink into node_modules/npm somewhere). No one would want to run an extensionless file with a flag anyway; node –entry-type=module npm doesn’t make sense (if you’re going to type all that, npm can just have an extension). So requiring executables like npm to be symlinks or inside package scopes, which most of them are already, doesn’t bother me; nor does it bother me to ask shell scripts to do the same or to use .mjs. Those solutions, I feel, are sufficient for those rare use cases.

Even with --input-type, whatever its final name becomes, there’s still the issue that people will probably expect/want it to work on files, and if it should work on files, then it should behave like --package-type. I still feel that way; but I’m willing to ship --input-type for now and wait for that feedback. When users complain that --input-type doesn’t work on files, we can ask them why they want it to; what are their use cases, beyond the two I listed above? If they’re compelling enough, that might lead us to support a flag for files, either like --package-type or maybe somehow different. But I see the logic that maybe we should wait for such feedback rather than building a flag we don’t know the use case for just yet.

So how would people feel about this approach? Replace --entry-type with --input-type now, and --input-type is what ships with Node 12; and we wait for feedback before expanding its scope further?

[targos]

I like this approach

[ljharb]

Given that being able to control the parse goal of non-file input is the only one of the three that is strictly necessary as opposed to being about convenience, sugar, etc, I wholeheartedly agree with this plan. It will neatly sidestep concerns about defaults, modes, consistency with package.json, etc; it makes something impossible possible; and it leaves open the design space to make something possible perhaps be simpler.

[tjcrowder]

Input from a user (me :-) ):

1. Glad to see --entry-type go, it was definitely confusing.

2. --input-type sounds great for the case it handles (direct input).

3. I really want --package-type as well, so that when throwing together a quick example I can use ESM with normal (to me) filenames and not have a package.json. I know it can be solved with a script (like Andrea's), but it seems like central functionality to me. Moreover, with clear semantics ("--package-type does what type in package.json does"), I don't see a strong argument against adding it. Yes, the bar to adding flags should be high-ish, but something as fundamental as ESM justifies it (to me, as a user).

Thanks to all for your deep thought, and hard work, on this. I'm overbooked the next two months, but I hope to start giving back by pitching in on things a newbie can help with (so, probably not this) in a couple of months.

[ljharb]

Why is it a need to not have a package json when using multiple files? Do you often have multiple files, no package.json (and thus no non-builtin dependencies), and run things directly with node?

[tjcrowder]

@ljharb - (Replacing my earlier reply, hit send too soon.) My quick examples are usually just a single file, but I prefer ESM to require even for built-in modules. But yes, sometimes they're a couple of files as well, and I'd want to use ESM to connect them.

[ljharb]

With a single file, you can use the proper extension (mjs) and it will be interpreted as such; at the moment the intention is for multiple files to require a package.json if you want to use an alternative file extension.

[tjcrowder]

@ljharb - "the proper extension" can be a heated phrase. For me, the proper extension is .js, and .mjs was a necessary temporary evil that these new features are making optional. Let's not turn this into a discussion on file extensions, it wouldn't be useful.

[tjcrowder]

I meant to say, and should have said, "can be a heated phrase," not "is". Sorry about that. (Didn't want to just silently edit.)

[ljharb]

Definitely let’s not debate it; but the reality is that i used an accurate term. It’s totally fine if you want to use a different extension, and i fully support (#283) everyone’s full freedom to do so - but that doesn’t change what the (not temporary) defaults are, and that you need a package.json to override the defaults.

[tjcrowder]

I'd use "default," it avoid the problems with "proper." The reality is indeed the .mjs is the default extension for ESM on Node.js.

...and that you need a package.json to override the defaults.

Or, ideally, --package-type. :-)

[bmeck]

--package-type still has some confusion due to it not being clear about override boundary being the cwd, all the fs, etc. Perhaps we can use a different name like --default-types , the usage of multiple formats covered by these flags also makes me wish we would use the plural

…

On Sat, Apr 13, 2019, 10:30 AM T.J. Crowder ***@***.***> wrote: I'd use "default," it avoid the problems with "proper." The reality is indeed the .mjs is the *default* extension for ESM on Node.js. ...and that you need a package.json to override the defaults. Or, ideally, --package-type. :-) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#300 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOUo2HIah-TZ7HV4QWzlGd_3wZzoFZUks5vgff_gaJpZM4cKuhd> .

[GeoffreyBooth]

One of the use cases discussed was wanting to change Node's default type, i.e. making .js treated as ESM by default when there's no package.json or there is one but it lacks a type field. That can be accomplished via --package-type but that's a counterintuitive way to achieve it. Perhaps a new flag just for this purpose, like --default-type=module that could be added to NODE_OPTIONS, could fit the bill.

@bmeck “types” already has meaning in TypeScript so it's probably not a good choice. I don't find the use of singular confusing, as the user is choosing only one type. And the browser attribute is just type.

[bmeck]

@GeoffreyBooth

The flag does not act similar to the browser attribute though and that at a glance leads to the confusion of treatment similar to how --entry-type was compared to the attribute in an odd way. The browser flag just uses an alternative scripting infrastructure and does not affect how the browser runtime identifies how to interpret content. The flag in Node does not cause a different scripting system to be used to load the module, it only affects how the ESM infrastructure identifies how to interpret a files contents for ESM.

The typescript argument also brings up verbiage problems (also see the previous issue that was brought) for dialogue that are being ignored here like:

"What type of package is X"
"X is a typescript package"
"No, what type does it export"
"X exports a Person class"
"No what is the content type like in the browser"
"X is type=module, but exports a text/javascript entrypoint"
"No, what is the type you set in package.json that node will get the content type from"
...
The term is very overloaded.

Also the flag can and should become general purpose due to needs of supporting other types such as .coffee, .ts, etc. for properly dealing with loader composition. We have had many talks about how the feature being described does not apply to a single type over time but must deal with further extension to handle existing types such as .jsx, .es6, etc. As well as portential new types.

None of these need to really be addressed when considering --default-types if we call it default-type , but the usage of singular is something we should probably avoid.