reflection: imperative equivalence for declarative import

[caridy]

as today, we only have loader.import('./foo.js').then(m => {...}), which is reflective to import * as m from "./foo.js", as we discussed earlier today, we might need to provide reflection for the other two options (default and named exports import declarations).

@dherman suggested:

local.import("./foo.js").then((def, named) => ...)
local.importNamespace("./foo.js").then(m => ...)

[domenic]

I really like the (def, named) => signature.

What is different between local.importNamespace("./foo.js").then(m => use(m)) vs. local.import((def, named) => use(named))?

[johnjbarton]

(Sorry for the meta discussion, but where I'm from "reflection" means functions for analysis, http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29.

This subject seems to be "Imperative equivalence for declarative loading" or similar).

[caridy]

To clarify, or to raise the question, depending on how you look at this, but local.import("./foo.js").then(def => ...) is really not reflective as it is the current proposal, essentially because def will not get updated if the value change in the provider module.

[domenic]

Hmm maybe that is an argument for only having the namespace form.

[guybedford]

importDefault?

[arv]

I'm not sure what problem we are trying to solve here?

How is

local.import("./foo.js").then((def, named) => ...)

better than:

local.import("./foo.js").then({default: def} => ...)
local.import("./foo.js").then({foo} => ...)

Lets try to keep the API surface area minimal. The special cases can always be done in user code and added later version of the spec.

[matthewp]

Yes, I'm with @arv on this, es6 with make this elegant enough without the need for multiple methods/signatures. loader.import should return the module. import/export is already complicated enough (I still have to reference this all the time), let's not muddle the dynamic api as well.

[dherman]

@arv @matthewp A low-level reflection API can be minimal and general. But the local import object is a high level API, meant for common use. And it's meant to be the dynamic analog of what you can do with the declarative syntax. It should have its defaults be symmetric with the surface syntax.

In the surface syntax, when you import a default, you don't have to write d-e-f-a-u-l-t. It's just the simplest thing where you have to say the least. In short, we want to allow:

local.import("./my-app-controller.js").then(MyAppController => ...)

as the most convenient case just like

import MyAppController from "./my-app-controller.js";

would be.

Another way of looking at this is that single-export needs to be as convenient in this API as it is today in existing JS module systems. Not having to say default explicitly is very important to the ergonomics.

But we also have to allow importing the namespace object, of course. And sometimes a module will not have a default export, so forcing you to write .then((dummy, named) => ...) is obnoxious for those cases. Hence a second method specifically for asking for the namespace object.

[dherman]

@caridy @domenic It's important to understand the use case of local import as not being about reflection but rather about dynamic loading. Reflection is a low-level operation meant for when you're trying to extend the base semantics of the language/system. Dynamic loading is simply a thing apps commonly need to do.

Also keep in mind that .import will not resolve the promise until the module has been initialized, fully executing its top-level code. The overwhelming majority of the time, the aliasing behavior is only important for the initialization phase where top-level bindings are being set up. It's extremely rare to mutate top-level exports after that point. So it's inappropriate to optimize this API around that as a constraint.

Again, the low-level loader API should fully generalize the semantics, and there it is certainly sensible to have operations that just produce the namespace object as the most general thing. But the purpose of the local import API is as a high-level API whose ergonomics and style should mirror the ergonomics and style of the declarative syntax.

[caridy]

Not having to say default explicitly is very important to the ergonomics.

Yes

It's extremely rare to mutate top-level exports after that point.

Agree. We can probably make the case that the value of the default export should never change, in which case something like local.import("./foo.js").then(def => ...) should be fine.

[matthewp]

You have to say default all the time already, it's a common way to export things. I see your point about having symmetry between the forms. But the declarative form is nice because of syntactical affordances you won't have here.

Question,

local.import("./foo.js").then((def, named) => ...)

even valid? I thought Promise only had a single return value.

[dherman]

You have to say default all the time already, it's a common way to export things.

The important distinction is that you only say default inside the implementation of a module, just as you have to do something explicit in CommonJS/Node (module.exports = ...;). But on the consumer side, you don't have to think about it having a name. Anyway sorry to beat a dead horse; it sounds like you agree the symmetry is important.

I thought Promise only had a single return value.

LOLOLOL. (At least I'm slightly comforted by the fact that the lead of the promises spec didn't catch this either. ;-P)

I think a reasonable alternative is two forms, with the short name providing the default, and the longer name providing the namespace object:

l.import("./rubber-chicken.js").then(RubberChicken => ...)
l.importAll("./utils.js").then(utils => ...)

The latter lets you extract the default export if you want:

l.importAll("jQuery").then(namespace => {
  let $ = namespace.default;
  ...
});

Or you can combine the two operations:

Promise.all(l.import("jQuery"), l.importAll("jQuery"))
  .then(([$, namespace]) => ...);

[caridy]

jejeje, importAll resonates well with me considering the work we have done on https://github.com/estree/estree/blob/master/es6.md#exportalldeclaration, which is symmetrical to the import all concept. I like it.

still we should clarify whether and how people will be able to update the default export from within a module, if there is no way to do that, then this syntax will be perfect.

[johnjbarton]

I think a reasonable alternative is two forms, with the short name providing the default, and the longer name providing the namespace object:

l.import("./rubber-chicken.js").then(RubberChicken => ...)
l.importAll("./utils.js").then(utils => ...)

I would appreciate a simplified explanation of what you all are talking about here. What is a "namespace object"?

[dherman]

What is a "namespace object"?

Also known as a module instance object in earlier versions of the module spec; ES6 now calls them "module namespace exotic objects." For example when you do

import * as fs from "fs";

It's the object that fs is bound to. It exposes all the module's named exports as properties (and if there's a default export, it has a named property called default that points to that).

[johnjbarton]

Thanks!

So the idea here is that

l.import("./rubber-chicken.js").then(RubberChicken => ...)
l.importAll("./utils.js").then(utils => ...)

is better than

l.import("./rubber-chicken.js").then(RubberChicken => ...);
l.import("./utils.js").then(ns => ns.default).then(utils => ...);

(where my import returns the namespace)?

IMO import should give the namespace. If we really need a thing for default (which we've almost never used in our es6 code), then it should have the special name, eg something obvious like importDefault. The short name should do the non-special thing.

[dherman]

An overloaded API like

l.import("./rubber-chicken.js").then(RubberChicken => ...);
l.import("./utils.js").then(ns => ns.default).then(utils => ...);

is bad because it changes what it returns based on whether you have a default export. But adding a default export should be a backwards-compatible API evolution: it should not break clients who aren't currently using it. The module system was designed with this principle as well.

IMO import should give the namespace. If we really need a thing for default (which we've almost never used in our es6 code), then it should have the special name, eg something obvious like importDefault. The short name should do the non-special thing.

I recognize there will be different styles in different communities, but default export is a highly popular pattern in many many JS ecosystems (from jQuery and underscore to AMD to NPM), and the ES6 module system picked that as a winner. In particular, the ergonomics of the import syntax in the module system are optimized for default export. The ES6 design isn't going to change, and the design of the reflective API has to parallel the design of the syntax.

[johnjbarton]

An overloaded API like

l.import("./rubber-chicken.js").then(RubberChicken => ...);
l.import("./utils.js").then(ns => ns.default).then(utils => ...);

There is no overloading here. The namespace is returned, always.

default export is a highly popular pattern in many many JS ecosystems (from jQuery and underscore to AMD to NPM), and the ES6 module system picked that as a winner.

That's quite different from my read. To me this default thing was added to appease the backward compat crowd. Really it seems obvious: if the ES6 module system picked it as winner then we would not need the funky extra default keyword at all!

Our dynamic import should mirror our static import:

  Reflect.loader.import('./foo.js').then(foo => ...);  // namespace foo

mirrors

  import * as foo from "./foo.js";

and

  Reflect.loader.importDefault('./foo.js').then(foo => ...); // default from foo

mirrors

  import { default as foo } from './foo.js';

[domenic]

To counterbalance @johnjbarton, in my ES6 projects I almost never use named imports.

I am still not very comfortable with the fact that .import() gives a non-updatable binding while import gives the magic-updating one. I would be more comfortable if we forced people to type .default since then the exotic object [[Get]] would be triggered each time, ensuring the binding stays updated.

[johnjbarton]

@domenic Could you point to an example? (Mine is the Traceur source https://github.com/google/traceur-compiler).

Is there a cogent pro/con discussion on this issue?

[caridy]

@johnjbarton nobody will do import { default as foo } from './foo.js';, they will do import foo from './foo.js'; that's the whole point @dherman is trying to make, you should NOT have to use the keyword default to import the default export, and that should be the most common use-case.

[caridy]

To counterbalance @johnjbarton, in my ES6 projects I almost never use named imports.

same here, you can check the of the packages used in formatjs.io, we use named exports in some places, but the majority of them are default exports.

[ericf]

While I do like the looks of the API that @dherman is proposing being the dynamic form of the importing the default export, I want to echo @domenic's concern about the non-updating binding. To me, this difference is trumping the economics argument and I think local.import() should return a Promise for the "module namespace exotic object."

My gut feeling is that the number of dynamic imports within an app's code base will be dwarfed by those which use the declarative syntax, so I don't think the economics arguments holds up against the wtf-js one.

[matthewp]

I'm still slightly on the side of only having the one signature import that returns the module. I do not see the benefit of only getting part of a module.

Regardless please do not choose importAll if you go this route. Users will assume it takes an array of modules to import. "import all of these things" makes sense, "return all of the bindings from this import" less so.

Personally I'm with @johnjbarton and use mostly named imports. But why should we base this on conjecture? It seems like there is enough es6 code floating around that a decent survey could be done.

[matthewp]

I think we might be over-estimating how used import is going to be. In my experience we use it in 2 or 3 places primarily:

1. To initially load your application. This will be superseded by <module>.
2. To progressively load parts of an application.
3. To conditionally load modules (based on environment variables for example).

1 is going to go away, and 2 and 3 are not used by a lot of applications anyways, and when they are it is used sparingly. I think import is safely within the realm of sophisticated applications. People working on these applications can handle default, imo.

[johnjbarton]

@caridy Nevertheless, any explanation of import foo from './foo.js'; will have to include the word default. To distinguish dynamic import of the namespace (that is, default thing we get without the default keyword) from the dynamic import of the developer-designated default we need a modifier. The natural modifier is 'Default', applied to obtain the developer-designated default. Applying a modifier "All" and having it return the thing we get by default without using default just compounds the confusion, in much the same way as I have done here.

[dherman]

Yikes, I believe I've perpetrated a pretty big confusion here. (Mea culpa!) There are two separate APIs to discuss, and I think at various points in this thread we may have been talking about one or the other. For clarity, let me give them names:

• reflective import: an API on Reflect.Loader.prototype that provides a complete reflection of what the module system can do
• async import: an API on the object bound via import local from this; that provides a way to progressively, asynchronously import modules

The difference

My view is that a reflective layer is meant more for advanced, frameworky code that is thinking at the level of the mechanics of the language semantics, but async import is more common. @matthewp fairly makes the point that it's still pretty rare. As @ericf put it in a private chat, it'll be common per-app-codebase but rare in frequency; most apps will have it, but each app will typically have only a handful of instances of it.

Constraints

Reflecting export mutability

We all agree it's a requirement for the low-level reflective API to reflect the mutability of exports. Module namespace objects are live views of the module's exports, so in that sense it's reasonable for the low-level reflective API to just have a single method that produces the namespace object.

Respecting the single-export mental model

The design of the module system allows a mental model where you can conflate the concept of a module's default export with the module itself. For example, you can create a UI component called AdminPanel that is the single export of a module, and allow the consumer to think of "admin panel" as being both the module and its sole export:

import AdminPanel from "./admin-panel.js";
...
someComponent.click()
  .then(() => {
    let panel = new AdminPanel();
    ...
  });

Notice how both the name of the module and the name of its exported abstraction are the same concept.

Now, if the programmer decides the admin panel is used rarely, they can do progressive loading of the code and change this to an async import. If we provide a convenient method for dynamic loading of default export, the programmer writes:

import local from this;
...
someComponent.click()
  .then(() => local.import("./admin-panel.js"))
  .then(AdminPanel => {
    let panel = new AdminPanel();
    ...
  });

If we only have a method that produces a namespace object, they have to drop below that level of abstraction:

import local from this;
...
someComponent.click()
  .then(() => local.import("./admin-panel.js"))
  .then(({ AdminPanel: default }) => {
    let panel = new AdminPanel();
    ...
  });

IOW, instead of the local import being just an async way of importing, it's also a reflection mechanism: you have to think about the details of the JS module system instead of just thinking about your app concepts.

Hazard of late mutations of default export

If the admin panel decides to mutate its default export late at runtime, it's possible for uses of a default-export-only API to go stale. For example, the admin panel might do something like:

var AdminPanel = class { ... };
...
// mutate the default export at runtime
someComponent.click().then(() => {
  AdminPanel = ...;
});
...
export { AdminPanel as default };

and then code that uses it might hang onto a stale binding:

...
local.import("./admin-panel.js")
  .then(AdminPanel => {
    someOtherComponent.click().then(() => {
      let panel = new AdminPanel(); // oops! didn't get the updated binding
    });
  });

Note that this is not an issue for mutation of the default export during initialization, since the import promise is only fulfilled after initialization completes. So for example you won't witness incomplete initialization of cycles. They have to explicitly modify the binding later on.

Consistency between the two layers

If both loader and local objects have importing APIs, for them to be subtly different would be confusing. For example, if loader.import() produces a namespace object but local.import() produces the default export, this would be an inconsistency between two APIs that are highly related.

My conclusion

I think this comes down to how real we think the risk of the hazard is and how bad the mental model violation is. IMO, the risk is low because people rarely mutate their exports, and the mental model violation is very bad for a high-level API, because the whole purpose of the design of the module system was to provide syntactic sugar to empower the conflation of single exports with their module. All that said, I recognize @matthewp's point that this is not going to be a ton of code per codebase, because progressive loading of an app is typically only done in a few select places.

So my preference is that we support both .import and .importAll on both objects. This allows you to get at the full namespace object and to get the full live-updated version of the default export if you have to. The worst-case scenario is that some popular library or libraries to export a runtime-mutating default export and it becomes a best practice to avoid .import and use .importAll, but I think it's better as a best practice to say don't mutate your exports after module initialization.

update: cleaned up some of the promise code in the examples

[caridy]

+1 on *.import() and *.importAll() where:

• importAll() is reflective of import * as ...
• and import() is just sugar on top of importAll() as in *.importAll("./mod.js").then(({ def: default }) => def).then(def => {...}); since def is not a live binding, but a stale binding.

[matthewp]

Fair points, I won't fight this too hard even though I don't love it.

Now back to bikeshedding :-) I really think All is wrong here. If a function is a verb and its arguments the object, importAll says "import all of these modules". But what we really are saying is "import everything from this module". importFrom maybe? If its named importAll I guarantee you devs will confusingly do:

importAll(['./login-panel.js', 'admin-panel.js']).then(function(panels) { });

[caridy]

@johnjbarton no. async import reference to the ability to access a loader instance that is bound to a module, and therefore it can perform the exact same import operations that you can achieve using the declarative syntax. We don't know yet, but in previous discussions it seems that it aligns well with what we want to achieve with the global loader, in other words, they will have the same API, although the internals are a little bit different since one of them is relative to a page, the other is relative a the module.

[guybedford]

How about importNS here? For import namespace. Would that terminology be correct?

[dherman]

It's hard to debate this until #27 is answered. From my perspective the spec authors haven't decided yet if they want Loader to be low-level and follow Extensible Web (which would be my preference) or be a high-level API that accomplishes only what is needed to load ES6 modules. From the issues I've participated in sometimes it seems they want it to be the former and sometimes the latter.

I certainly don't feel undecided about it! :) I tried to answer the question in this thread. In short:

• The static syntax is the overwhelmingly common case and of course high-level.
• Async import, while less common than syntax, will still be likely to come up a few times in most any reasonable-sized app, and is a high-level API.
• Reflective import will be used rarely, mostly in the internals of loading frameworks. It's a low-level API.

@matthewp Async import and reflective import are two different APIs, and they don't have the same design constraints. In particular, async import is tied to syntax so it can never be fully abstractable. This means people are more prone to suffering through the papercuts.

Maybe we need to just admit that ES6 doesn't have a single export and not tailor the API to the hack to make it (almost) appear that it does.

You're eliding the semantic model with the programming model. While the internal semantics of ES6 does not have any special concept of a default export, default export is a central part of the design. It's part of the way modules are thought about and the way they are written.

I appreciate people working through the different possibilities here, btw. I know it can be frustrating but IMO it's always good to map out a space of possibilities before landing too soon on a conclusion. In that spirit, another comment coming in a sec... ;-P

[dherman]

@johnjbarton No, import local from this; is a candidate syntax for functionality we've known has been missing from ES6: the ability to expose to each module its own specific information. This includes metadata like its filename as well as the ability to do dynamic loading relative to that module.

[dherman]

@domenic @caridy Let's hold off on the overloading question as a mostly-orthogonal question for just a bit.

[dherman]

So here's another syntactic possibility we hadn't considered previously for async import. :D

A late-breaking addition to ES6 was the new.target syntax, and in the process of discussing the idea, Allen Wirfs-Brock mentioned the possibility of exposing module metadata via an analogous import.foo syntax. The attractive thing about this is that it allows programmers to get access to metadata and async import without needing to inject an extra import declaration at top-level. (This was a benefit of my original idea of using this, but @domenic -- rightly! -- objected that we should not encourage the use of this outside of the context of methods and class constructors.)

In that light, a candidate syntax-and-API for async import would be:

import.default("Spinner").then(Spinner => { ... });
import.namespace("fs").then(({ readFile, writeFile }) => { ... });

(I can live with making default explicit here for a few reasons. I'll grant @matthewp's point that this is more advanced than the declarative syntax. But it's still important to have convenient async API to match the convenient syntax because, as I've said, it's exposed to syntax and higher level than the reflection API. And it's important for the default case to be more convenient than the namespace case.)

Note that if we specify .default and .namespace to be hard-bound to this module, then they can conveniently be extracted and used in higher-order abstractions. So even without overloading, the convenience of Promise.all is pretty good:

Promise.all(["Spinner", "Slider", "Gallery"].map(import.default)).then(...);
Promise.all(["fs", "util", "collections"].map(import.namespace)).then(...);

[dherman]

Forgot to mention, I also propose the metadata (like filename, URL, debugging info, etc) live as a separate subobject, probably import.metadata. It's nice to separate the concerns of async import from accessing module metadata, and it also sets the metadata object aside as the place where host environments can stash non-standard properties.

[matthewp]

@dherman Can you explain how this works? Is import a contextual object available within a module?

[dherman]

@matthewp It's not reflected as a first-class object, in that you can't get your hands on the object directly. It's like an object where the syntax only allows you to access certain predetermined properties. So for example you can ask for new.target but you can't say var x = new; ... x.target ...

[dherman]

But yes, it's contextual per-module.

[matthewp]

That's interesting, I hadn't heard of this feature. I'll have to dive into esdiscuss then, but I think I understand what you're saying.

I like the idea overall and agree it's a clever way of working around this problem. You do still want access to the loader within a module, import.local maybe.

[dherman]

I should also show what this ends up looking like in async functions:

let Spinner = await import.default("Spinner");
let { readFile, writeFile } = await import.namespace("fs");
...

[matthewp]

Very elegant!

[MajorBreakfast]

This look pretty elegant!
Everything suddenly becomes very clean if we can go this way.

Don't be too frustrated: We can restart the naming discussion around this syntax! :) :)

Promise.all(['Spinner', 'Slider', 'Gallery'].map(import)).then(...) // Default import, call import as a function

import.namespace('fs').then(fs => ...) // Namespace import

import('fs', 'readFile').then(readFile => ...) // New idea: Named import shortcut

You tell me if treating import as a function does even work. I just assume that it's possible since the static syntax always includes the from part, which should enable parsers to tell the two apart. Parsers are probably able to tell the two apart because the static import is never valid ES5 and this is except for the reserved keyword thing. Not sure if we want to go this way, but I'm proposing it anyway :)

[bmeck]

Does this need to be a metaproperty and not some form of a function? It seems like a language extension would mean this loader would have to be the backer for all host environments if it assumes it gains meta-properties.

[matthewp]

@bmeck Not having the Loader be the backer for all host environments would be a major mistake.

[bmeck]

@matthewp you need to explain that in depth. loader vs es import/export are 2 different things to my knowledge

[matthewp]

They are different things but users write code that run in multiple JavaScript environments.

[bmeck]

@matthewp it has different implications semantically than existing code/systems (couchdb, webworkers, mongodb, node, ringojs, ...) which is the issue. having the semantics able to match would be nice as well as be much safer for upgrade purposes

[matthewp]

I agree, a breaking change for those legacy systems is probably a no-go. I wish there were more participation here from all interested parties but this repo has been largely dark for the last several months.

There's a specification at amodro-lifecycle that attempts to address some of these legacy needs such as Node, I would suggest giving it a read if you have time.

[bmeck]

@matthewp my last experience here did not include metaproperties or requirements for it to be used as the import hook handler for VMs. WebAssembly/design#256 brought me here made me see several changes like this issue which make it a much larger scope than I thought it was. my interest came as a result of the increased scope and WASM making me come reread it in the current form

[matthewp]

The metaproperty proposal only exists in this issue and hasn't been written to any spec AFAIK. There hasn't been any increase in scope to my knowledge, it is now much smaller is scope actually.

[bmeck]

@matthewp we commented on it because it is a language extension which immediately concerned us. A lot of work is often done in conversations and only shows up in specs to replace TODOs etc. at a very late stage if we want to read public available text. I think talking in issues is fairly similar to talking about the spec.

[caridy]

@matthewp @bmeck just a side note, the import imperative call is bound to the loader instance that loaded the host module in the first place (just like require() is bound to the host module), which means all modules will have access to it since they all have a loader associated, even if it is the built-in default loader implemented by the runtime.