ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING error when using ESM script to render

[aKzenT]

When trying to use a ESM script with the library I'm getting a ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING error on the line that dynamically imports the script.

A dynamic import callback was not specified. TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified. at new NodeError (node:internal/errors:399:5) at importModuleDynamicallyCallback (node:internal/modules/esm/utils:89:9) at Http2ServerRequest. ([eval]:1:3021) at Http2ServerRequest.emit (node:events:512:28) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
at Jering.Javascript.NodeJS.HttpNodeJSService.TryInvokeAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.TryInvokeCoreAsync[T](InvocationRequest invocationRequest, CancellationToken cancellationToken)
at Jering.Javascript.NodeJS.OutOfProcessNodeJSService.InvokeFromFileAsync[T](String modulePath, String exportName, Object[] args, CancellationToken cancellationToken)

This is the line from the Javascript.NodeJS library minified script that causes the exception:

      a = e.endsWith(".mjs") ? await import("file:///" + e.replaceAll("\\", "/")) : require(e)

The script with "mjs" extension looks very simple:

export default async(data) => {
  console.log("render called");
  return Promise.resolve("Test");
};

The strange thing is also that initially it does seem to work, so some calls are successful, but then it stops working with that error.

Any ideas?

I'm using node 19.7.0 and 7.0.0-beta3 of this library.

[JeremyTCD]

It's not your logic, I noticed this error in a recent CI build.

This library's logic is doing the bare-minimum, just calling import(<file>). From that and the intermittent nature of the issue, it looks like a Node.js problem.

I've found where the error is thrown in Node.js: https://github.com/nodejs/node/blob/0c460518e8c0728294cfbb5313d347a62856a2c5/lib/internal/modules/esm/utils.js#L89.

When I have some time to investigate I will look into it, but it may be a while (several weeks).

If you have spare bandwidth, it would be helpful if you could try creating a repro of the issue (without this library) and reporting the issue to the Node.js team if you manage to.

In the meantime, .mjs should not be used in production with this library.

[aKzenT]

I did try some simple reproductions by trying to import a mjs module from a script evaluated dynamically from the command line like the library does. However even when calling import multiple times, I did not get that error. So I have no idea what exactly triggers this and it does not seem to be a known problem in NodeJS.

One simple workarround that works for me is just wrapping the MJS file in a CJS file like this:

module.exports = async(data) => {
    var render = (await import('./render.mjs')).default;
    return await render(data);
};

One idea you might want to look into is changing the embedded scripts to be ESM as well and then specify the --input-type on the command line (https://nodejs.org/dist/latest-v19.x/docs/api/cli.html#--input-typetype). My limited understanding is that in general it is easier to import a CJS file into ESM than the other way arround.

[JeremyTCD]

Okay, "script evaluated dynamically" meaning using --eval/-e?

Do you mind cloning this repo, adding a test that reproduces the issue and creating a PR? Would be good to have a test case to debug against.

I'm open to converting the embedded scripts to es modules, but will need some way to verify that doing so fixes the issue.

[aKzenT]

I created a PR that reproduces the issue fairly reliable on my PC. However it seems that this only happens infrequently. So I had to do a lot of retries which makes the test very slow.

One thing I noticed in the test protocol though was that even before the exception occurs there are connection resets and retries with the error stating that the connection was closed by the remote host. That makes me wonder whether maybe the error does occur every time, but the library is not fast enough to read the error before the process crashes. Could that be possible?

[JeremyTCD]

Thanks for the PR.

With regard to the "connection closed" retries, dynamic imports are within a try-catch block. If the dynamic import error gets caught some of the time, I don't see how it could get past the try-catch block on other occasions.

Imo it's likely some other error, but might be related? Either fatal (killing the process immediately) or uncaught error. I've just done some reading up, we should add:

process.on('uncaughtException', ...);
process.on('unhandledRejection', ...);

to rule out uncaught errors. Should also check windows event viewer for any further information on fatal errors.

Another thing we can do would be to simplify the HTTP server to its most basic (for invoking .mjs), see if that resolves the issue and add back stuff till we find the offending logic. For example, right now the server polls the .Net process every second (Shared.ts), stuff like that might be affecting the server.

If none of that resolves the issue, we'll have to look into Node.js, crash dumps, etc. Appreciate if you could start on investigations, will try when I have more time!

[thebuilder]

Was just wondering if this was just me. Also getting the error after switching our Vite output to the modern .mjs.
Great that you are already on top of this. 💪 Vite 5 might drop support for .cjs, so getting it working properly for the future would be nice: vitejs/vite#12466 🙏

A dynamic import callback was not specified.
TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.
    at new NodeError (node:internal/errors:371:5)
    at importModuleDynamicallyCallback (node:internal/process/esm_loader:39:9)
    at IncomingMessage. ([eval]:1:3020)
    at IncomingMessage.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

[thebuilder]

Hey @JeremyTCD - Have you had a chance to dig further into this issue?🙏

One simple workarround that works for me is just wrapping the MJS file in a CJS file like this:

module.exports = async(data) => {
    var render = (await import('./render.mjs')).default;
    return await render(data);
};

Have you been using this workaround in production @aKzenT? Any issues?

[JeremyTCD]

Hello Daniel, I must apologize, as I haven't had the opportunity to work on this library recently due to other commitments. I'm looking to bring in new maintainers to help manage the project. If you, or anyone else, are interested in this role, please don't hesitate to let me know.

…

Message ID: ***@***.*** com>

[aKzenT]

Have you been using this workaround in production @aKzenT? Any issues?

So far I did not have any issues with this approach, but it is not in production yet. That being said, I did encounter the issue even during development when using ESM directly, so I'm feeling confident that the workarround works for now. A native solution would of course be nicer.

[thebuilder]

So far I did not have any issues with this approach, but it is not in production yet. That being said, I did encounter the issue even during development when using ESM directly, so I'm feeling confident that the workarround works for now. A native solution would of course be nicer.

Looked like a good solution, but I did a quick test of this approach, and ran into this error in my project:

import { hyphenateHTMLSync } from "hyphen/da/index.js";
         ^^^^^^^^^^^^^^^^^
SyntaxError: Named export 'hyphenateHTMLSync' not found. The requested module 'hyphen/da/index.js' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

The error is triggered by loading .cjs, that loads .mjs that then loads another .cjs from node_modules. I did the test by loading the file directly with Node, so not even caused by this library.

Since Vite 4.4.4, they are logging a warning that .cjs support will be removed in version 5.

UPDATE: Solved the issue by changing the import, so it imports the .js file directly. 👍

[thebuilder]

I'm not that confident i .NET, but trying to get the project to run, and looks like it needs some env variables to run the Webpack build task. Right now it fails because env.mode and env.entry is undefined.

[aKzenT]

Unfortunately I just encountered this issue on production even with the workarround of loading ESM from CJS :-(

So it seems that ESM will not work correctly until the loader script is in ESM as well. @thebuilder did you figure out how to build the project?

[thebuilder]

Unfortunately I just encountered this issue on production even with the workarround of loading ESM from CJS :-(

So it seems that ESM will not work correctly until the loader script is in ESM as well. @thebuilder did you figure out how to build the project?

Which issue? I solved the CommonJS imports issue, by changing the import (to what it suggests in the error mesage).

We are currently looking into running pure Node server alongside the .NET project, and having them communicate over localhost HTTP.

[aKzenT]

I mean I got ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING exceptions even when using a CJS wrapper to import my ESM file.

[thebuilder]

Looking at the JavaScript part of the project - Thinking it would make sense to replace the current basic Webpack bundle, with a Vite/Rollup solution. Vite will most likely handle the import correctly in the bundle it makes.

• Replace Webpack with Vite
• Change the package.json type to module
• Bundle the Http11Server and Http22Server to ESM

Modules are supported since Node 14 (currently the oldest supported version).
Still struggling with actually running/testing this, but I can share the changes I did if it makes sense.

[aKzenT]

I just created a PR to move the project to ESM. All unit tests are green and my own project also is able to run as before. I noticed @thebuilder you had the same idea and even switched to vite, which is a great idea. However I don't think your PR is complete as there are some pieces missing to make the whole pipeline ESM.

Feel free to incorporate my changes in your branch and we can wait for for @JeremyTCD to decide if he is okay to switching to vite or if he wants to stay with webpack for now.

What issues do you have with running the unit tests? Maybe I can help. I also created a PR to fix some unit tests problems I encountered.

[thebuilder]

I just created a PR to move the project to ESM. All unit tests are green and my own project also is able to run as before. I noticed @thebuilder you had the same idea and even switched to vite, which is a great idea. However I don't think your PR is complete as there are some pieces missing to make the whole pipeline ESM.

Feel free to incorporate my changes in your branch and we can wait for for @JeremyTCD to decide if he is okay to switching to vite or if he wants to stay with webpack for now.

What issues do you have with running the unit tests? Maybe I can help. I also created a PR to fix some unit tests problems I encountered.

Cool - I'll do that.
Issue with getting it to run, might be because I'm using a Mac with Rider. I'm missing some of the .NET frameworks it expects.

[JeremyTCD]

Thanks for the prs, I don't have access to a pc till Friday, so will merge them then. Moving to esm looks good. About webpack > vite, my (admittedly superficial) understanding is that vite's designed to serve sites on localhost during development, and uses rollup under the hood when you build for production. Do we just use rollup in our case?

…

On Wed, 26 Jul 2023, 19:51 Daniel Schmidt, ***@***.***> wrote: I just created a PR to move the project to ESM. All unit tests are green and my own project also is able to run as before. I noticed @thebuilder <https://github.com/thebuilder> you had the same idea and even switched to vite, which is a great idea. However I don't think your PR is complete as there are some pieces missing to make the whole pipeline ESM. Feel free to incorporate my changes in your branch and we can wait for for @JeremyTCD <https://github.com/JeremyTCD> to decide if he is okay to switching to vite or if he wants to stay with webpack for now. What issues do you have with running the unit tests? Maybe I can help. I also created a PR to fix some unit tests problems I encountered. Cool - I'll do that. Issue with getting it to run, might be because I'm using a Mac with Rider. I'm missing some of the .NET frameworks it expects. — Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZQXCTZAPURH4U7XVI3DP3XSEADXANCNFSM6AAAAAAVTTZ5KY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[thebuilder]

Vite is pretty flexible. It can be used to serve SPA applications, but it also has built in support for SSR - So using this, we can have it output files that are already optimized for Node rendering.
You could use Rollup, but then you'd have to figure out the right configuration settings/plugins.

With Vite, this command is all that's really needed, to bundle the TypeScript for Node.

vite build --ssr ./Servers/OutOfProcess/Http/Http11Server.ts --outDir ./bin/Debug/

[aKzenT]

I was trying out my changes on Azure App Service and encountered an issue on how webpack transforms import statements. Basically it uses "import.meta.url" which is 'undefined' due to loading the script as a string instead of as a file. I'm not sure why it works locally. This might be due to a different node version.

The vite code does not have this issue and is generally cleaner and much closer to the original code. So I would also suggest to use vite.

I will update my PR to use vite based on your changes @thebuilder . Hopefully I can then do some more tests until Friday to confirm that the issue is gone for good.