Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for JS macros #9299

Merged
merged 15 commits into from
Jan 6, 2024
Merged

Add support for JS macros #9299

merged 15 commits into from
Jan 6, 2024

Conversation

devongovett
Copy link
Member

@devongovett devongovett commented Oct 8, 2023

This adds support for importing and executing JS macros during the build. It follows the same API as implemented in Bun, using import attributes to determine which modules are macros.

import { test } from "./macro.js" with { type: "macro" };

const staticValue = test(1, 2);

These are then evaluated inside the JS transformer by calling back from Rust into JS, requiring the dependency, and running the function. Arguments are statically evaluated, then serialized to napi values.

export function test(a, b) {
  return a + b;
}

Macros return JS values which are then converted back into an AST and inserted where the call was. The output of the above code would be:

const staticValue = 3;

The dependency is not included in the final bundle. This is useful for computing values during the build that might be expensive in terms of bundle size to do at runtime, or to insert values beyond simple environment variables. Macros also run before the dead code eliminator, so if the result of a macro is used in an if statement the dead branch can be removed.

To do

  • Support Bun's "macro" package.json exports condition?
  • Track statically known variable values so they can be used in arguments?
  • Add a way to return raw JS source code from macros in addition to values? Would be nice to allow macros to return a generated function for example.
  • Support macros written in TypeScript? Would mean transpiling in the package manager...

@parcel-benchmark
Copy link

parcel-benchmark commented Oct 8, 2023

Benchmark Results

Kitchen Sink ✅

Timings

Description Time Difference
Cold 1.47s +28.00ms
Cached 276.00ms +17.00ms ⚠️

Cold Bundles

Bundle Size Difference Time Difference
dist/legacy/index.ff03421b.js 1.48kb +0.00b 381.00ms +32.00ms ⚠️
dist/legacy/index.e9bb1616.js 1.06kb +0.00b 381.00ms +33.00ms ⚠️
dist/modern/index.4a29d309.js 921.00b +0.00b 381.00ms +33.00ms ⚠️
dist/legacy/index.html 826.00b +0.00b 417.00ms +31.00ms ⚠️
dist/modern/index.html 749.00b +0.00b 417.00ms +32.00ms ⚠️

Cached Bundles

Bundle Size Difference Time Difference
dist/legacy/parcel.7cdb0fad.webp 102.94kb +0.00b 242.00ms -43.00ms 🚀
dist/legacy/parcel.7cdb0fad.webp 102.94kb +0.00b 243.00ms -43.00ms 🚀
dist/modern/parcel.7cdb0fad.webp 102.94kb +0.00b 243.00ms -43.00ms 🚀
dist/legacy/index.ff03421b.js 1.48kb +0.00b 364.00ms -26.00ms 🚀
dist/legacy/index.e9bb1616.js 1.06kb +0.00b 364.00ms -26.00ms 🚀
dist/modern/index.4a29d309.js 921.00b +0.00b 364.00ms -26.00ms 🚀
dist/legacy/index.html 826.00b +0.00b 364.00ms -55.00ms 🚀
dist/modern/index.html 749.00b +0.00b 363.00ms -56.00ms 🚀
dist/legacy/index.b8ae99ba.css 94.00b +0.00b 252.00ms -41.00ms 🚀
dist/modern/index.31cedca9.css 94.00b +0.00b 251.00ms -42.00ms 🚀

React HackerNews ✅

Timings

Description Time Difference
Cold 3.86s +41.00ms
Cached 405.00ms +5.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/PermalinkedComment.e9dc4a75.js 3.92kb +0.00b 336.00ms +25.00ms ⚠️
dist/UserProfile.8945a243.js 1.38kb +0.00b 336.00ms +25.00ms ⚠️
dist/NotFound.8b44a81d.js 269.00b +0.00b 336.00ms +25.00ms ⚠️
dist/logo.8dd07848.png 244.00b +0.00b 263.00ms +41.00ms ⚠️

Cached Bundles

Bundle Size Difference Time Difference
dist/PermalinkedComment.e9dc4a75.js 3.92kb +0.00b 340.00ms +27.00ms ⚠️
dist/UserProfile.8945a243.js 1.38kb +0.00b 340.00ms +27.00ms ⚠️
dist/NotFound.8b44a81d.js 269.00b +0.00b 340.00ms +27.00ms ⚠️

AtlasKit Editor ✅

Timings

Description Time Difference
Cold 31.35s -456.00ms
Cached 2.15s +64.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/media-viewer.38e3999a.js 536.13kb +0.00b 7.38s -2.72s 🚀
dist/archive.c374f622.js 59.90kb +0.00b 7.38s -2.81s 🚀
dist/media-viewer-analytics-error-boundary.60bdaa4c.js 3.18kb +0.00b 7.38s -2.81s 🚀
dist/ru.0cf3f40e.js 2.81kb +0.00b 5.17s -2.14s 🚀
dist/codeViewerRenderer.51140ec8.js 2.61kb +0.00b 7.38s -2.81s 🚀
dist/heading5.d2f94d9d.js 1.23kb +0.00b 4.63s -654.00ms 🚀
dist/ro.8d5b380a.js 482.00b +0.00b 5.17s -2.14s 🚀
dist/index.html 248.00b +0.00b 4.41s -5.96s 🚀

Cached Bundles

Bundle Size Difference Time Difference
dist/pdfRenderer.01deafa1.js 12.04kb +0.00b 7.33s -2.67s 🚀
dist/heading3.73972e33.js 1.35kb +0.00b 5.21s +624.00ms ⚠️
dist/heading2.c27d912d.js 1.17kb +0.00b 5.21s +624.00ms ⚠️
dist/pt_PT.e211e609.js 635.00b +0.00b 5.28s -1.54s 🚀

Three.js ✅

Timings

Description Time Difference
Cold 2.83s +32.00ms
Cached 310.00ms +6.00ms

Cold Bundles

No bundle changes detected.

Cached Bundles

No bundle changes detected.

Click here to view a detailed benchmark overview.

@mischnic

This comment was marked as resolved.

@mischnic
Copy link
Member

mischnic commented Nov 6, 2023

Another API question here is how to control caching. In your example that's not a problem but a useful macro would be getting the current git commit, current date, ....

@mischnic mischnic marked this pull request as draft November 13, 2023 15:04
@devongovett
Copy link
Member Author

Added a few new features:

  1. Ability to return functions from macros. Dynamically created functions can be created with new Function. The source of the function is then parsed with SWC and inlined into the AST. Perhaps not the most efficient because it'll be parsed twice (by node as well), but this doesn't require any custom API.
  2. Support for macros written in TypeScript. This is done by running @swc/core inside NodePackageManager, so it would actually also allow TS plugins and dev dependencies as well (but not when inside node_modules).
  3. Ability to emit additional assets from a macro. For example, a macro could emit CSS to go along with the JS it produces. Currently the API for this is an additional (hidden) argument passed as the macro's this. That way it doesn't appear in TS types from the outside. TS macros can type the this as void | MacroContext and check if this exists to know if they are being called as a macro.

The question about caching is a good one. Right now the code for the macro or the code calling the macro would need to change for it to be invalidated (like a normal dev dependency). Perhaps the macro context (this) could have additional methods in the future to control caching. Not sure exactly what the options should be – invalidateOnStartup, invalidateOnBuild, etc?

@devongovett
Copy link
Member Author

Would it be unexpected for the results of macros to be hoisted out as constants? For example if a macro returned a function/object, we know that value will never change, and cannot depend on inputs since it was evaluated statically. Might be nice to hoist it to allow things like react to optimize re-renders because the value isn't re-created each time. However, this is different from the normal semantics of JS, so maybe it would be unexpected?

@devongovett devongovett marked this pull request as ready for review January 1, 2024 01:26
@mischnic
Copy link
Member

mischnic commented Jan 1, 2024

Would it be unexpected for the results of macros to be hoisted out as constants?

Probably yes. Especially if you mutate the result (like array and objects) of the macro call which would break with hoisting.

However, if the return value is indeed an immutable "primitive" value (so everything except object/array), then it should be possible to hoist (ignoring that you can of course assign properties to functions).

We could also have two modes for the macro expansion, without and with hoisting. {type: "macro", hoisted: true} or something like that

Not sure exactly what the options should be – invalidateOnStartup, invalidateOnBuild, etc?

Yeah, could it just be all of the invalidations we already have on Config/MutableAsset?

@devongovett
Copy link
Member Author

Looks like invalidateOnStartup and invalidateOnBuild only exist in Config right now and not MutableAsset. Adding those will be much easier after we drop the transformation cache as previously discussed, otherwise I'd have to add something to the cache key. I pulled that out into a separate PR: #9459

Comment on lines 30 to 36
let b = await bundle(path.join(dir, '/index.js'), {
inputFS: overlayFS,
mode: 'production',
});

let res = await overlayFS.readFile(b.getBundles()[0].filePath, 'utf8');
assert(res.includes('output=3'));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha nice. updated them to use a string hash function instead.

@devongovett
Copy link
Member Author

Added functions to the macro context to control caching. This also resulted in invalidateOnStartup and invalidateOnBuild being added to MutableAsset and some internal refactoring to the way we store invalidations internally.

export function test(a, b) {
return a + b;
}
import { hashString as foo } from "@parcel/rust" with { type: "macro" };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice that you can import any package here, I wouldn't have thought of trying that 👍

@devongovett devongovett merged commit f5ebdd3 into v2 Jan 6, 2024
15 of 16 checks passed
@devongovett devongovett deleted the macros branch January 6, 2024 21:05
lettertwo added a commit that referenced this pull request Jan 30, 2024
* upstream/v2: (22 commits)
  Add source map support to the inline-require optimizer (#9511)
  [Web Extension] Add content script world property to manifest schema validation (#9510)
  feat: add getCurrentPackageManager (#9505)
  Default Bundler Contributor Notes (#9488)
  rename parentAsset to root for msb config and remove unstable (#9486)
  Macro errors -> v2 (#9501)
  Statically evaluate constants referenced by macros (#9487)
  Multiple css bundles in Entry bundle groups issue (#9023)
  Fix macro issues (#9485)
  Bump follow-redirects from 1.14.7 to 1.15.4 (#9475)
  Revert more CI changes to centos job (#9472)
  Use lightningcss to implement CSS packager (#8492)
  Fixup CI again (#9471)
  Clippy and use napi's Either3 (#9047)
  Upgrade to eslint 8 (#8580)
  Add support for JS macros (#9299)
  Fixup REPL CI (#9467)
  Drop per-pipeline transformation cache (#9459)
  Upgrade some CI actions (#9466)
  REPL (#9365)
  ...
@mischnic mischnic mentioned this pull request Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants