Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: define es6 package as es6 package #5635

Merged
merged 7 commits into from
Sep 18, 2023
Merged

fix: define es6 package as es6 package #5635

merged 7 commits into from
Sep 18, 2023

Conversation

epreston
Copy link
Contributor

@epreston epreston commented Sep 14, 2023

The source (not the build products) for the project are es6 modules. Inform all tools that this is an es6 project.

At the moment, the package.json does not specify this correctly so tools assume "cjs". Some entry points are marked with "mjs" to correct this, but, there will always be a new edge case where incorrect assumption is uncovered.

In other words, this sets the "source" assumption to be "modules", so "js" is assumed to be "mjs", "cjs" can still be used to mark tooling code that needs to be interpreted as node only "cjs". "mjs" is reserved for node only code that needs to be evaluated as a module.

Renaming all the files to .mjs is pointless once the project is marked correctly.

References: #5623
References: #5555

Fixes the odd amount of effort expended juggling file extensions for js files.

I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

@mvaligursky
Copy link
Contributor

Looks good to me.
Any thoughts @kungfooman ?

@mvaligursky mvaligursky self-assigned this Sep 14, 2023
@epreston
Copy link
Contributor Author

epreston commented Sep 14, 2023

To remain consistent with standards:

"cjs" - node only code written in commmonjs
"mjs" - node only code using es6 modules
"js" - browser and tooling code (node independent) written to the package standard

The package.json directs a few things for consumers of the library. There's a few of odd naming conventions and configuration settings in place to "make things work" because the package type was not set. I'd like to talk about build products and simplify this in another PR.

@kungfooman
Copy link
Collaborator

Looks good to me too, even though I don't have the time right now to check every consequence... but that will come by itself.

Wouldn't we need a scripts/package.json to "unset" them from being ESM now?

About renaming every engine file to .mjs... I basically just like the idea because in some contexts you simply cannot quickly judge the content otherwise without peeking into the file (and even then it could be a wrong assumption) + e.g. file filtering in VSCode would be more precise.

@epreston
Copy link
Contributor Author

Does not look like a "package.json" is required for them.

Only question mark is opentype and earcut in the scripts/textmesh folder, everything is either an IIFE (which works everywhere) or just simple javascript in a declarative es5 style (which works everywhere).

If they are "node only commonjs", they should be named cjs to avoid bundling them for browsers by mistake.

Using the correct application of "cjs", "mjs" and "js" keeps things very clear. Marking browser code as node code, in order to know what is es5 and es6 is odd.

@epreston
Copy link
Contributor Author

Update: finished checking /scripts. text-mesh is already an es6 class using assuming a global of "earcut" is defined. I guess that is not used by the es5 editor. The rest don't show any issues. I don't see anything stopping this code from being executed in an es5 or es6 environment.

@kungfooman
Copy link
Collaborator

Can you explain how you test the changes here? You talk about "tools", but which tool breaks and how do you notice that the breakage is fixed with the changes here - to make it obvious what we are dealing with here

Since you define type=module in /package.json, I guess we can delete this file too?

{
"type": "module"
}

@epreston
Copy link
Contributor Author

epreston commented Sep 15, 2023

Happy to help, its a big topic there's a few perspectives so help me narrow this down. I'll write out my main thoughts and motivation.

Yes, that is not required and can be removed as well.

@epreston
Copy link
Contributor Author

epreston commented Sep 15, 2023

My approach to tool testing:

I maintain public template projects on github for all the major bundlers under this account. (vite, webpack, parcel, variant configurations of rollup) I quickly spin them up to ensure a dependency can be consumed gracefully.

I maintain a public template project specifically for playcanvas under this account to test how the library is consumed by 3rd parties.

@epreston
Copy link
Contributor Author

epreston commented Sep 15, 2023

The package.json is the traffic director for tools, a source of meta for a project, we can also cram other tool configurations in there.

Through the type, types, exports, main, module, sideEffects properties we communicate the expectations of what 3rd parties can consume. These third parties are bundlers, people, editors, and common tools. You kind of build a support grid from here. (I want to write out the grid, happy to do it)

In that grid, this setting communicates to:

  • node import
  • bundlers consuming source
  • people using the library
  • editors resolving symbols

Through the type property specifically, we communicate the expectations of the source code used to create the build products. Which is ES modules.

This will be essential in divesting "extras" from core code and having a good developer experience.

When its set correctly on each project dependency in your app, you don't think about what is cjs, es modules, iife, you just import and it works.

The build tools in this project (and the majority of the popular build tools) actually transform everything to ES modules before processing. Sometimes just to transform them back, sometimes to ensure they align to the requirements of the build product.

Setting it correctly leaves room for overrides by file extension. The overrides become redundant on a modern project, so, "mjs" and "cjs" can be used as a distinction between "browser" client code and "node" tooling code. Since only node needs the "cjs" extension, it gets earmarked with extensions.

Sometimes its just nice to ignore the package.json to force a CJS import for constants or plain javascript for a mesh decimation tool. Requires extra steps if you lock in mjs.

This project is less than an handful of of files away from living in a ES2022 world. Literally a weekend and 3 PRs. The only meaningful file extension for "source" after the next few weeks is "js" vs "ts".

I agree, this has more meaning in projects which have more of a mixture of frontend, backend, and CLI. Seeing as some PC projects sometimes assume they are all cloned into the same parent folder as Engine, it's a valid concern.

@kungfooman
Copy link
Collaborator

@epreston Thank you for your feedback, currently examples/rollup.config.js is using a file extension check:

entries: {
'playcanvas': ENGINE_PATH,
'../../../build/playcanvas.js': ENGINE_PATH,
'pc-alias': ENGINE_PATH.includes('.mjs') ? './pc-es6.mjs' : './pc-es5.mjs'
}

That implies you cannot do ENGINE_PATH=../src/index.js right now.

Now every problem has a bunch of different solutions... and testing for .mjs is:

  1. easy to implement, harder to introduce bugs or increase complexity where it's not needed
  2. quicker for both human and machine - skipping extra parsing of JSON files

But yes, I also see your point... in a way I like to keep everything *.js aswell... after all we can already do:

ret = await import("http://127.0.0.1/playcanvas-engine/build/playcanvas.js")

Important difference is:

image

And this difference can then be used to check if it was ESM or not... in case of ESM, we have to do window.pc = ret2; to support both UMD and ESM.

You talked about this article yesterday: https://2ality.com/2016/11/trace-globals-proxy.html

And I'm not sure how that relates to importing or how that helps, because it is eval-ing a string, so it doesn't "hook" into the import system (to get accessed globals for assigning them to window?).

That would never be the case and difficult design to justify. You can't load cjs to begin with in the browser. Commonly people on this project confuse "iife" and "cjs". You can consume "iife" directly from ES modules. There is no additional thought or effort.

Right, everything is confusing 😅 And iife is unified in umd, what the build/playcanvas.js is, therefore we can consume it via await import(url). I also saw you talking about ES6 flavor in regards to ESM yesterday, which is also not the same: we had ES6 const/let/classes way before ESM import/from/export (can't find your comment any longer to check again).

Anyway, I think this discussion was already fruitful and I will remove some dependence on MJS file extension checking in #5555 while not introducing more code... not sure if I can do so 100% yet.

@epreston
Copy link
Contributor Author

epreston commented Sep 15, 2023

That's a check on build products. It's not related to this. No limitations are introduced.

In a world where file extensions are omitted and people have more than a handful of dependencies in their projects, nobodies managing importmaps by hand to ship products.

There is no need to support UMD for any platform or use case. It's the full technical equivalent of IIFE with less compatibility and no defined standard. Delivering an IIFE supports the oldest legacy and ESM in one shot.

About the ES6 proxies, examples, editor: I'm writing up a discussion combining your ideas with what I see.

no longer required
Reserve extensions for marking node code, not client code.
Update root file
@kungfooman
Copy link
Collaborator

I retract my LGTM after 738d19d

I know your stance and all, but it doesn't make sense to me. I've attempted to explain the reasoning behind the JS/MJS change multiple times, but it appears you haven't quite understood yet or you simply don't want to see the issue.

@epreston
Copy link
Contributor Author

You want to use the most granular definition when it could be reserved, leaving space for it to be ignored. It's an odd measure aligned to managing importmaps by hand.

@willeastcott
Copy link
Contributor

My feeling is that we should consider parking the JS->MJS rename for now and allow @epreston to run with this for a bit. We can reassess the rename later - I mean, it was just a casual suggestion from me in the first place. Unless there's something blocking the merging of the rewritten Examples Browser?

@kungfooman
Copy link
Collaborator

Unless there's something blocking the merging of the rewritten Examples Browser?

I liked it for making one decision as simple as it gets, but I can add some code to figure out the difference at build time if ENGINE_PATH is UMD or MJS, so the Examples browser can consume both again (build outputs and various ESM index.js from different engine PR's and quick testing etc.).

@epreston
Copy link
Contributor Author

Need some help ?

playcanvas.js should be the IIFE.
playcanvas.mjs should be the es module build

The current examples browser does not consume source, it consumes build products, it also contains an example of making the distinction.

@kungfooman
Copy link
Collaborator

The current examples browser does not consume source, it consumes build products, it also contains an example of making the distinction.

It's funny that you want to unify UMD and ESM so hard, but then you have such a hard opinion about build products vs. source.

It's both... consumable source code. One main difference is that you need to invest work to create a build product, whereas source is free.

So the price of a build product is:

  1. You need to handle multiple open shells for npm run build/watch
  2. Many extra resources / file operations on your disc
  3. The rollup processing in itself
  4. The cognitive overhead to simply get started

That's all wasting time and energy for nothing. When I'm testing a PR or do bug fixing, I do none of that. I use source for the quickest possible workflow. This allows you to work as fast as possible, time is money etc. 🤑

Need some help ?

Thank you for asking, I pretty much just need a node/fs function right now that reliably tells me if a file is UMD or EJS if you want to work it out already. I don't know how much time you have at hand, but I have other things on my TODO list aswell 😅

@epreston
Copy link
Contributor Author

epreston commented Sep 16, 2023

I don't want to unify anything. There needs to be multiple build products. UMD is an abandoned pattern that hasn't become a standard. It has drawbacks, incompatibilities with the last 8 years of standards, and you can ship 2 other formats covering all use cases, never having an issue. Mix and match them, etc.

The source is at a ES11 language level, trying to consume it directly as ES5 or make any distinctions like that won't go very far. One reason for the build system is to translate that down on at least one build product to the lowest common support target which is ES5-ish. This is why engine build products are consumed by the examples app to create an ES5 build.

I understand the confusion and complexity. Things can be simplified; external requirements are forcing some extra machinery.

@epreston
Copy link
Contributor Author

To answer your question:

You'll already know from the build file name what you are importing. Importing either incorrectly won't build. Still..

Off the top of my head, if an extra check at runtime is required for some reason, the node function you are looking for is the humble "import" statement; the same why you would when you want "side effects" from a polyfill.

import './file.js';

It works on both modules and IIFEs. If the global variable "pc" is defined (the side effect), you imported the IIFE. If not, you have the module.

@kungfooman
Copy link
Collaborator

Need some help ?

playcanvas.js should be the IIFE.
playcanvas.mjs should be the es module build

@epreston I don't have a sense we speak the same language here, but thank you for trying to help, I will implement my requirements myself.

I'm not sure why you bring up IIFE all the time now, maybe you are confusing that with UMD. UMD literally means Universal Module Definition and 100 % encompasses IIFE - that's why it's called universal 💡

And our ES5 build/playcanvas.js build is UMD and not IIFE.

@epreston
Copy link
Contributor Author

No worries.

The code you have linked and reviewed is for selecting build products from the build directory. Should be safe to use the existing code "as is" for selecting between an ES5 or modular build. It has nothing to do with source.

Read up on UMD. Having universal in the name hasn't helped it. A UMD is an IIFE with less compatibility.

There are a few issues and omissions with the build products. That's one of them.

@kungfooman
Copy link
Collaborator

Read up on UMD. Having universal in the name hasn't helped it. A UMD is an IIFE with less compatibility.

Wrong. A IIFE literally just wraps your code in a function to spare you from leaking global module variables into globalThis. It doesn't attempt to make sure your code name (pc) is added into globalThis. UMD does, while doing everything that IIFE does (wrapping your code so it doesn't leak anything).

I wrote:

And our ES5 build/playcanvas.js build is UMD and not IIFE.

You wrote:

playcanvas.js should be the IIFE.

Response: You can't admit that you were wrong once.

@epreston
Copy link
Contributor Author

Dig into it when you have time.

@kungfooman
Copy link
Collaborator

Dig into it when you have time.

I don't know whom you are trying to fool here, but anyway:

engine/rollup.config.mjs

Lines 288 to 291 in 3032623

const outputFormat = {
es5: 'umd',
es6: 'es'
};

I'm out of this PR.

@epreston
Copy link
Contributor Author

epreston commented Sep 16, 2023

A UMD is a pattern of writing an IIFE with a few extra if statements

Here it is in it's full form.

// myModuleName.js
(function (root, factory) {
    if (typeof define === 'function' && define.amd) {
        // AMD. Register as an anonymous module.
        define(['exports', 'b'], factory);
    } else if (typeof exports === 'object' && typeof exports.nodeName !== 'string') {
        // CommonJS
        factory(exports, require('b'));
    } else {
        // Browser globals
        factory((root.myModuleName = {}), root.b);
    }
}(typeof self !== 'undefined' ? self : this, function (exports, b) {
    // Use b in some fashion.

    // attach properties to the exports object to define
    // the exported module properties.
    exports.action = function () {};
}));

AMD and commonjs are useless in the browser space. The library does not run in node, AMD needs a runtime to work in a browser. These extra if statements make it break if used in ESM.

You are pointing to a mistake in the build configuration.

@willeastcott
Copy link
Contributor

I just looked at three and they seemed to have marked their UMD build as deprecated: https://github.com/mrdoob/three.js/blob/dev/utils/build/rollup.config.js#L154

@epreston
Copy link
Contributor Author

epreston commented Sep 16, 2023

Yep. It was useful in 2016, today, not so much with a supported standard built into the language and browsers. It's a liability if someone uses the "import" statement with the wrong file.

Some people like the global namespace with everything in it, one file, instead of a collection of them. Same as this project does with some files in the scripts folder. A plain iife would allow that.

You do the chart and you get the same support matrix. Correction, it adds support for an extra use case.

@willeastcott
Copy link
Contributor

Soooo.....we mergin'? 😄

@willeastcott willeastcott merged commit 88c6028 into playcanvas:main Sep 18, 2023
7 checks passed
@epreston epreston deleted the define-source-type branch September 18, 2023 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants