Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust v1.82 enables reference type transformations by default #4211

Open
ctm opened this issue Oct 18, 2024 · 20 comments
Open

Rust v1.82 enables reference type transformations by default #4211

ctm opened this issue Oct 18, 2024 · 20 comments
Labels

Comments

@ctm
Copy link

ctm commented Oct 18, 2024

Describe the Bug

The Hello World! example fails to run or even compile with Rust 1.82.0, presumably due to reference types. That example uses webpack with a configuration that will invoke @webassemblyjs/wasm-parser which doesn't support reference types. This other bug report has a backtrace that is similar to what happens now.

Steps to Reproduce (when using Rust 1.82.0)

  1. cd examples/hello_world
  2. install the needed node packages (I used yarn install, I'm sure npm has something similar, possibly npm install)
  3. Try to build (e.g. npm run build or yarn build)
  4. See backtrace

Everything you need is in this repository other than installing the npm modules.

Expected Behavior

[main]% rustup default 1.81.0
rustup default 1.81.0
info: using existing install for '1.81.0-x86_64-apple-darwin'
info: default toolchain set to '1.81.0-x86_64-apple-darwin'

  1.81.0-x86_64-apple-darwin unchanged - rustc 1.81.0 (eeb90cda1 2024-09-04)

[main]% npm run build
npm run build

> build
> webpack

🧐  Checking for wasm-pack...

✅  wasm-pack is installed at /Users/ctm/.cargo/bin/wasm-pack. 

ℹ️  Compiling your crate in development mode...

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling hello_world v0.0.0 (/Users/ctm/git_others/wasm-bindgen/examples/hello_world)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.43s
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: ✨   Done in 0.87s
[INFO]: 📦   Your wasm pkg is ready to publish at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/pkg.
✅  Your crate has been correctly compiled

assets by status 54 KiB [cached] 1 asset
asset index.js 16.3 KiB [emitted] (name: main)
asset index.html 232 bytes [compared for emit]
runtime modules 4.87 KiB 7 modules
cacheable modules 3.61 KiB (javascript) 54 KiB (webassembly)
  modules by path ./pkg/*.js 3.5 KiB
    ./pkg/index.js 141 bytes [built] [code generated]
    ./pkg/index_bg.js 3.36 KiB [built] [code generated]
  ./index.js 48 bytes [built] [code generated]
  ./pkg/index_bg.wasm 70 bytes (javascript) 54 KiB (webassembly) [built] [code generated]
webpack 5.95.0 compiled successfully in 1640 ms

Actual Behavior

[main]% rustup default stable
rustup default stable
info: using existing install for 'stable-x86_64-apple-darwin'
info: default toolchain set to 'stable-x86_64-apple-darwin'

  stable-x86_64-apple-darwin unchanged - rustc 1.82.0 (f6e511eec 2024-10-15)

[main]% npm run build
npm run build

> build
> webpack

🧐  Checking for wasm-pack...

✅  wasm-pack is installed at /Users/ctm/.cargo/bin/wasm-pack. 

ℹ️  Compiling your crate in development mode...

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling hello_world v0.0.0 (/Users/ctm/git_others/wasm-bindgen/examples/hello_world)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.37s
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: ✨   Done in 0.80s
[INFO]: 📦   Your wasm pkg is ready to publish at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/pkg.
✅  Your crate has been correctly compiled

assets by status 1.58 KiB [cached] 1 asset
asset index.js 13.2 KiB [emitted] (name: main)
asset index.html 232 bytes [compared for emit]
runtime modules 1.77 KiB 4 modules
cacheable modules 3.91 KiB (javascript) 56.1 KiB (webassembly)
  modules by path ./pkg/*.js 3.82 KiB
    ./pkg/index.js 167 bytes [built] [code generated]
    ./pkg/index_bg.js 3.66 KiB [built] [code generated]
  ./index.js 48 bytes [built] [code generated]
  ./pkg/index_bg.wasm 40 bytes (javascript) 56.1 KiB (webassembly) [built] [code generated] [1 error]

ERROR in ./pkg/index_bg.wasm
Module parse failed: Unknown element type in table: 0xNaN
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
Error: Unknown element type in table: 0xNaN
    at parseTableType (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1018:13)
    at parseTableSection (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1274:24)
    at parseSection (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1406:24)
    at Object.decode (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/decoder.js:1740:25)
    at decode (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/@webassemblyjs/wasm-parser/lib/index.js:253:21)
    at WebAssemblyParser.parse (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/wasm-async/AsyncWebAssemblyParser.js:61:19)
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:1303:19
    at processResult (/Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:937:11)
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/webpack/lib/NormalModule.js:1030:5
    at /Users/ctm/git_others/wasm-bindgen/examples/hello_world/node_modules/loader-runner/lib/LoaderRunner.js:407:3
 @ ./pkg/index.js 1:0-40 4:15-19 5:0-21
 @ ./index.js 1:0-30 3:0-5

webpack 5.95.0 compiled with 1 error in 1532 ms

Additional Context

I believe this can be fixed by disabling the reference-types feature for wasm32-unknown-unknown, however even after reading the Enabled WebAssembly Features section of the rustc book (and trying various things), I have not been successful at disabling that feature.

@widavies
Copy link

widavies commented Oct 18, 2024

I second this, cannot get this working on latest versions

Rust: 1.82
Webpack: 5.95.0
wasm-bindgen: 0.2.95

Trying to add a wasm rule in my Webpack config just leads to different errors:

  module: {
    rules: [
      {
        test: /\.wasm$/,
        type: "asset/resource"
      }
    ]
  },
ERROR in ../pkg/index.js 5:0-21
Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)
 @ ./index.js 1:0-43 4:17-29
 @ ./bootstrap.js 4:0-20

@ctm
Copy link
Author

ctm commented Oct 19, 2024

Upon further reading and experimenting, I suspect that for now it will simply impossible to use 1.82.0 with anything that uses wasm-pack, because disabling reference-types requires unstable features and 1.82.0 itself is stable. If so, this demo won't be able to use a stable 1.82 or later rust release until they either make it so that reference-types is not enabled by default or provide a stable way (that works through the wasm-pack stack) to disable that default.

@nickbabcock
Copy link
Contributor

Similarly, Rust 1.82 and Node 18 don't play well together when running wasm-bindgen-test, as node will crash with a bit of a nasty stacktrace.

@CryZe
Copy link
Contributor

CryZe commented Oct 19, 2024

So there's multiple things at play here. LLVM activated the reference-types feature by default, which got pulled into Rust 1.82. The reference-types feature generally affects almost nothing in how LLVM generates code, except it now supports some extended table index encoding.

The problem is that wasm-bindgen looks at the features section of the module and assumes that you want FULL reference-types support. It does so here:

// Enable reference type transformations if the module is already using it.
if let Ok(true) = wasm_bindgen_wasm_conventions::target_feature(&module, "reference-types")
{
self.externref = true;
}

Usually you are meant to use the wasm-bindgen CLI flag --reference-types to achieve the same effect. But this automatic detection above activates it anyway now.

This by itself isn't really a problem, but webpack, or rather its webassemblyjs dependency is mostly unmaintained and does not currently support the reference-types proposal.

@daxpedda I propose that wasm-bindgen for the time being should be patched to not automatically turn on the externref support if it detects it from the features section. I can do that PR if you think this is the right course of action.

@Kofituo
Copy link

Kofituo commented Oct 23, 2024

I literally created this exact issue last 2 weeks and it was quickly dismissed. I am glad it's finally being worked on as this is a major issue.

@mplanchard
Copy link

mplanchard commented Oct 25, 2024

Running into the same issue here:

❯ webpack  --version 

  System:
    OS: Linux 6.6 NixOS 24.05 (Uakari) 24.05 (Uakari)
    CPU: (20) x64 13th Gen Intel(R) Core(TM) i7-1370P
    Memory: 21.40 GB / 62.52 GB
  Binaries:
    Node: 20.9.0 - /nix/store/igr9rd5rbpm9bagf49q20b9vhcag9w08-nodejs-20.9.0/bin/node
    Yarn: 3.2.0 - /nix/store/60izdbxbkzm0fkknwi357xi5lcszcwwi-yarn-1.22.19/bin/yarn
    npm: 10.1.0 - /nix/store/igr9rd5rbpm9bagf49q20b9vhcag9w08-nodejs-20.9.0/bin/npm
  Browsers:
    Firefox: 131.0.2
  Monorepos:
    Yarn Workspaces: 3.2.0

❯ rustc --version
rustc 1.82.0 (f6e511eec 2024-10-15)

❯ wasm-bindgen --version
wasm-bindgen 0.2.92

Using react 18 with CRA-generated config, applying the following overrides to get wasm to work:

    if (!config.resolve.extensions) {
      config.resolve.extensions = [];
    }
    if (!config.resolve.extensions.includes(".wasm")) {
      config.resolve.extensions.push(".wasm");
    }

    const wasmExtensionRegExp = /\.wasm$/;

    if (!config.module.rules) {
      config.module.rules = [];
    }

    // Ensure that file-loader ignores WASM files.
    // Sourced from https://prestonrichey.com/blog/react-rust-wasm/
    config.module.rules.forEach((rule) => {
      (rule.oneOf || []).forEach((oneOf) => {
        if (oneOf.loader && oneOf.loader.indexOf("file-loader") >= 0) {
          // Make file-loader ignore WASM files
          oneOf.exclude.push(wasmExtensionRegExp);
        }
      });
    });

    // Use wasm-loader for WASM files
    config.module.rules.push({
      test: wasmExtensionRegExp,
      // I'm going to level with you: I copied this in from the example, but I
      // have no idea why it's necessary. If it's not here, it breaks, though.
      include: path.resolve(__dirname, "src"),
      use: [{ loader: require.resolve("wasm-loader"), options: {} }],
    });

Definitely seems limited to webpack for the bundler target specifically. We use the same package compiled for the nodejs target in Node without issues, as well as in svelte via vite.

bidzyyys pushed a commit to OpenZeppelin/rust-contracts-stylus that referenced this issue Oct 28, 2024
It seems that reference types from WASM are not supported by
`wasm-parser` crate
(rustwasm/wasm-bindgen#4211).
It makes us pin to the recent nightly toolchain version that works, and
contribution experience gets quite convoluted.
Hopefully there is an option to disable `reference-types` with a flag:
`-C target-cpu=mvp` for `wasm32-unknown-unknown` target.

Resolves #374

Single source of rust toolchain from `rust-toolchain.toml` added for ci
pipelines (actions-rust-lang/setup-rust-toolchain@v1 gh action used for
that).

Resolves #294
@daxpedda
Copy link
Collaborator

@CryZe

This by itself isn't really a problem, but webpack, or rather its webassemblyjs dependency is mostly unmaintained and does not currently support the reference-types proposal.

@daxpedda I propose that wasm-bindgen for the time being should be patched to not automatically turn on the externref support if it detects it from the features section. I can do that PR if you think this is the right course of action.

My first gut feeling is to continue as-is. Unfortunately, I'm a really clueless about webpack and the ecosystem it sits in. But it seems to me, that if its unmaintained, users should upgrade to something maintained instead of the rest of the ecosystem having to stay behind for the purpose of compatibility.

Inherently, this is a problem that should be solved by Rustc and not by wasm-bindgen. The reality however is that this is a bigger problem in the current toolchain (WebAssembly/tool-conventions#233) and wasm-bindgen is not the part playing its role here. However, it seems to me that applying this fix in wasm-bindgen doesn't really fix the issue, because LLVM already emits reference-type proposal specific things (e.g. table encodings) that could affect any part of the toolchain not supporting this already.

In that light, I will go ahead and pin this issue so we can get as much input as possible before settling on a solution!

@daxpedda daxpedda changed the title Hello World! example breaks with Rust 1.82.0 Rust v1.82 enables reference type transformations by default Oct 29, 2024
@daxpedda daxpedda pinned this issue Oct 29, 2024
@CryZe
Copy link
Contributor

CryZe commented Oct 29, 2024

My PR as per my testing fully unblocks the webpack ecosystem. It would only be a short term bugfix to unblock people who have been affected by it. I'm slowing working on PR on supporting reference types fully in webassemblyjs / webpack, but it probably is still taking a while (also depending on how responsive the maintainer is). So I would recommend going with my PR until webpack is fixed, which hopefully shouldn't take too long (but probably in the range of multiple months).

@daxpedda
Copy link
Collaborator

While I wouldn't mind holding off until then, I would like to know how this is going to play out in the future. What do we do when LLVM enables other proposals by default which break webpack? It seems unsustainable to me.

ls it possible to move wasm-pack away to something more sustainable?
If the issue only affects the bundler target in practice, could we change the behavior only for that target?

@mplanchard
Copy link

Just speaking for us as an example, we have a large, legacy application react using webpack. We would love to be able to move off of it, and we are in the process of doing so, but it is going to take some time. It would be great to not have to pin our Rust compiler to an old version until that happens, so a temporary fix would be helpful, whether that be here, in a webpack plugin, or wherever.

@Kofituo
Copy link

Kofituo commented Oct 30, 2024

Just speaking for us as an example, we have a large, legacy application react using webpack. We would love to be able to move off of it, and we are in the process of doing so, but it is going to take some time. It would be great to not have to pin our Rust compiler to an old version until that happens, so a temporary fix would be helpful, whether that be here, in a webpack plugin, or wherever.

This seem to work for now
webpack/webpack#15566 (comment)

@mplanchard
Copy link

mplanchard commented Oct 30, 2024 via email

@mplanchard
Copy link

FYI the strip = true solution doesn't seem to work for me.

Added the following in the package's Cargo.toml

[build]
strip = true

[profile.dev]
strip = true

[profile.release]
lto = true
opt-level = 'z'
strip = true

but still get the following when trying to import the WASM file in the built application:

Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)

@Kofituo
Copy link

Kofituo commented Oct 30, 2024

FYI the strip = true solution doesn't seem to work for me.

Added the following in the package's Cargo.toml

[build]
strip = true

[profile.dev]
strip = true

[profile.release]
lto = true
opt-level = 'z'
strip = true

but still get the following when trying to import the WASM file in the built application:

Can't import the named export '__wbindgen_start' (imported as 'wasm') from default-exporting module (only default export is available)

This sample repository works
https://github.com/Kofituo/hello-world/tree/master

@mplanchard
Copy link

Yeah, I believe this could be due to some settings in create-react-app? Our current override is to include wasm files and load them with the wasm-loader plugin, which still results in this error. I haven't found another way to override yet to get it working with the stripped WASM binary.

@daxpedda
Copy link
Collaborator

I went ahead and implemented the proposal from #4211 (comment) in #4235: reference type proposal transformations aren't enabled by default for the bundler target.

Please let me know if this addresses at least the majority of use-cases out there.

@ctm
Copy link
Author

ctm commented Oct 31, 2024

Works for our code. Thanks!

@ctm
Copy link
Author

ctm commented Oct 31, 2024

I apologize for the premature false positive. Our stack was giving 0 exit status and ending with a lot of lines of output that looked right, even though there was a panic:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.36s
thread 'main' panicked at /Users/ctm/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-bindgen-cli-support-0.2.95/src/decode.rs:61:27:
mid > len
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Running the wasm-bindgen CLI
Caused by: Running the wasm-bindgen CLI
Caused by: failed to execute `wasm-bindgen`: exited with exit status: 101
  full command: "/Users/ctm/.cargo/bin/wasm-bindgen" "/Users/ctm/poker/rust/poker/target/wasm32-unknown-unknown/debug/mb2_web.wasm" "--out-dir" "/Users/ctm/poker/rust/poker/front-ends/web/pkg" "--no-typescript" "--target" "bundler" "--out-name" "index" "--debug"

After that follows 32 lines of webpack output showing webpack doing things as if everything was running fine, terminating with:

webpack 5.95.0 compiled successfully in 3930 ms

IOW, webpack didn't notice the error and stop what it was doing and exit with a non-0 status. I was in the middle of lunch but was (too) eager to let you know that it worked that I posted once I saw what I thought was a happy build.

Here's the output with a full backtrace:

[master]% time RUST_BACKTRACE=full yarn build
time RUST_BACKTRACE=full yarn build
🧐  Checking for wasm-pack...

✅  wasm-pack is installed at /Users/ctm/.cargo/bin/wasm-pack. 

ℹ️  Compiling your crate in release mode...

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
    Finished `release` profile [optimized + debuginfo] target(s) in 0.27s
thread 'main' panicked at /Users/ctm/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-bindgen-cli-support-0.2.95/src/decode.rs:61:27:
mid > len
stack backtrace:
   0:        0x101afc193 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::haf41a54e7ed5fe07
   1:        0x101b18e1b - core::fmt::write::h526161fad96c5ad5
   2:        0x101af975e - std::io::Write::write_fmt::h4de0398536c8b0e4
   3:        0x101afd2fa - std::panicking::default_hook::{{closure}}::hffeef89117b5d267
   4:        0x101afcfaa - std::panicking::default_hook::h2f20b53f3264c5d5
   5:        0x101afdfdf - std::panicking::rust_panic_with_hook::h7919e8c602131afa
   6:        0x101afd832 - std::panicking::begin_panic_handler::{{closure}}::h7bd628128f542977
   7:        0x101afc669 - std::sys::backtrace::__rust_end_short_backtrace::ha2fdb908654ed930
   8:        0x101afd47c - _rust_begin_unwind
   9:        0x101b4aa4a - core::panicking::panic_fmt::h292eaa4d42e451fa
  10:        0x101865ee3 - <&str as wasm_bindgen_cli_support::decode::Decode>::decode::h183f37fe2eef9785
  11:        0x101878c37 - <alloc::vec::Vec<T> as wasm_bindgen_cli_support::decode::Decode>::decode::hb82d6dcdb406ba77
  12:        0x101867699 - <wasm_bindgen_cli_support::decode::Struct as wasm_bindgen_cli_support::decode::Decode>::decode::ha4e303ee138df1ea
  13:        0x101877f4e - <alloc::vec::Vec<T> as wasm_bindgen_cli_support::decode::Decode>::decode::h35d1fb876e33758f
  14:        0x10186600a - <wasm_bindgen_cli_support::decode::Program as wasm_bindgen_cli_support::decode::Decode>::decode::hd0d3f5c90cd692eb
  15:        0x10183a0f5 - wasm_bindgen_cli_support::wit::extract_programs::h6ec4f369a111a7e9
  16:        0x101885280 - wasm_bindgen_cli_support::Bindgen::generate_output::h0c0d1ba5ed33744e
  17:        0x1017ebedb - wasm_bindgen::main::h0844e41c4e2b6a10
  18:        0x1017e99c6 - std::sys::backtrace::__rust_begin_short_backtrace::hc4d326f764bb84ff
  19:        0x1017e99bc - std::rt::lang_start::{{closure}}::hfa575cd405def99e
  20:        0x101af329f - std::rt::lang_start_internal::h8372644ac40c1e8b
  21:        0x1017ec46c - _main
  22:     0x7ff80b8382cd - <unknown>
Error: Running the wasm-bindgen CLI
Caused by: Running the wasm-bindgen CLI
Caused by: failed to execute `wasm-bindgen`: exited with exit status: 101
  full command: "/Users/ctm/.cargo/bin/wasm-bindgen" "/Users/ctm/poker/rust/poker/target/wasm32-unknown-unknown/release/mb2_web.wasm" "--out-dir" "/Users/ctm/poker/rust/poker/front-ends/web/pkg" "--no-typescript" "--target" "bundler" "--out-name" "index"

@daxpedda
Copy link
Collaborator

That is a different error entirely.
My first guess is a different CLI and wasm-bindgen version, but that should be caught by the hash check.
Could you try using Rust v1.81 just to make sure this is unrelated?

@ctm
Copy link
Author

ctm commented Oct 31, 2024

Thanks! Aha. Yes. I was inadvertently running an old wasm-bindgen:

[master]% which wasm-bindgen
/Users/ctm/.cargo/bin/wasm-bindgen

All I had done to get the wasm-bindgen crate was to have added

[patch.crates-io]
wasm-bindgen = { git = "https://github.com/rustwasm/wasm-bindgen" }

to my top-level Cargo.toml, but that doesn't bring in a new wasm-bindgen CLI. I didn't think about that because the stack I use is fairly opaque to me and I never run wasm-bindgen by hand; it's just done by magic when I say yarn dev or yarn build.

So now that I've also done a cargo install wasm-bindgen-cli --git "https://github.com/rustwasm/wasm-bindgen", my development and release builds work and build usable artifacts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants