Add support for ESM modules

[perrin4869]

Fixes #1014

Edit: oops clicked enter before writing the explanation
I tried out the jest-light-runner as a way to not break the jest test suite and enable using imports at the same time.
The unit tests are passing, but I am stuck right now due to nicolo-ribaudo/jest-light-runner#36
I will come back to this and try to iron it out once I figure that problem out, and add some tests

[perrin4869]

@dnalborczyk I see you've been making some refactoring of the repo that is related to this PR. I solved all conflicts and made sure the tests are passing. The only thing missing from this PR are some regression tests...

Edit: windows is failing ><;;

[perrin4869]

OK, fixed windows CI and added tests for running under type: module, I think this is ready now :)

[dnalborczyk]

@dnalborczyk I see you've been making some refactoring of the repo that is related to this PR. I solved all conflicts and made sure the tests are passing. The only thing missing from this PR are some regression tests...

Edit: windows is failing ><;;

sorry, was just reading this.

@perrin4869 could you hold on for just a moment. most of the refactoring (babel removal, esm-only-module) stuff is done, but there's a couple albeit minor things missing: logging needs a cleanup (not quite sure why yet, I believe serverless v3 added a different logging?) as well as re-activating unit tests (and some other few tests I deactivated). should be hopefully done soon.

[dnalborczyk]

@perrin4869 just had a quick look.

I believe a better way to implement this is something along the lines of:

• js extension without a package.json "type" should be loaded as commonjs (via createRequire/require)
• cjs extension "type" should be loaded as commonjs (via createRequire/require)
• mjs extension via import()
• js extensions with package.json type=commonjs via createRequire/require
• js extensions with package.json type=module via import
• probably also needs to follow a specific order of extension resolution. (in case js and cjs or mjs are also a handler in the same directory

you can take a look here: https://github.com/aws/aws-lambda-nodejs-runtime-interface-client the problem is that this is not the latest version which actually runs in a lambda container I believe, since esm support is missing in this repo, but working in the container.

I have to find the latest runtime someone extracted from the current lambda container.

[perrin4869]

Yeah, rather than Promise.any it's indeed a good idea to have a priority of extension resolution... ['js', 'mjs', 'cjs'] seems reasonable?
Is there a reason to separate all these cases though? import can resolve and load both commonjs and esm modules after all...

[perrin4869]

@dnalborczyk done!

[tinchoz49]

Hey @perrin4869, if main points to src/index.js the src directory should be added to the publish. https://github.com/perrin4869/serverless-offline/blob/feature/esm-imports/package.json#L52

Except for that, I tried the PR in my projects and it worked perfect. Very happy of removing webpack/babel from my serverless life. Good work!

[perrin4869]

@tinchoz49 oh nice catch! But that's a problem in the master branch ;)

[tinchoz49]

ah ok, something to change when was merged. Another thing is that in some files is using the old __dirname and throws me an error in my handlers.

[perrin4869]

if you really want this working right now, I recommend you use the patch in #1014 with patch-package, that's what I've been doing for many months now without problems. The change in this PR is a complete implementation with tests, etc

[perrin4869]

@dnalborczyk merged the latest changes in master and fixed the tests, will be happy if you can review this :)

[dnalborczyk]

thank you @perrin4869 ! it's the next feature on the list. give me a couple days, just need to release v9.1

[dnalborczyk]

@perrin4869 I just noticed that I added a code review but forgot to hit "send" (request changes). 😞

I added esm support as part of using the aws-lambda-ric UserFunction.js (which also includes namespace support): #1534

I hope by using UserFunction.js directly the functionality becomes more future-proof. unfortunately the repository https://github.com/aws/aws-lambda-nodejs-runtime-interface-client seems to be more or less unmaintained and is (currently) not up-to-date with the lambda runtimes and the lambda docker container (where this function has been extracted from). once the module has been updated again, we can just reference the package and remove the copy/pasted code.

thank you for putting in the work and spearheading the effort!

[perrin4869]

@dnalborczyk No, thank you for the effort in updating this module properly!
I tried your PR and it works wonderfully for loading ESM!
However, it is lacking one feature that this PR had, which is the ability to load ts files when --experimental-loader ts-node/esm is available (for example, by using NODE_OPTIONS="--experimental-loader ts-node/esm".
This can be easily added by simply adding this patch:

$ cat patches/serverless-offline+9.2.0.patch
diff --git a/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/aws-lambda-ric/UserFunction.js b/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/aws-lambda-ric/UserFunction.js
index 1542880..46c8cea 100644
--- a/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/aws-lambda-ric/UserFunction.js
+++ b/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/aws-lambda-ric/UserFunction.js
@@ -274,6 +274,7 @@ const { pathToFileURL } = require('node:url')
     const loaded =
       (pjHasModule && (await _tryAwaitImport(lambdaStylePath, '.js'))) ||
       (await _tryAwaitImport(lambdaStylePath, '.mjs')) ||
+      (await _tryAwaitImport(lambdaStylePath, '.ts')) ||
       _tryRequireFile(lambdaStylePath, '.cjs')
     if (loaded) {
       return loaded

I don't know if it is within the scope of this module to add the ability to load ts files, as this isn't an official feature in lambda, but I find this useful, and I think other people might as well.
I was thinking that maybe adding an extensions option to this plugin might be a good way to add support for something like this...
Any thoughts?

[dnalborczyk]

However, it is lacking one feature that this PR had, which is the ability to load ts files when --experimental-loader ts-node/esm is available (for example, by using NODE_OPTIONS="--experimental-loader ts-node/esm".

I was wondering what that was about. it's definitely interesting. I wonder if the ts-node loader approach even works without adding anything to this plugin? I have to give it a deeper look.