Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

guest-async component build error #1

Closed
xdlin opened this issue Jan 16, 2025 · 12 comments
Closed

guest-async component build error #1

xdlin opened this issue Jan 16, 2025 · 12 comments

Comments

@xdlin
Copy link

xdlin commented Jan 16, 2025

Hi @dicej I'm very interested in the async component function, thanks for your great work! When I tried to build it locally, but I have this error:

wasm-tools component new --adapt wasi_snapshot_preview1.reactor.wasm \  
    target/wasm32-wasi/release/guest_async.wasm -o guest-async.wasm

I got this error:

error: failed to encode a component from module

Caused by:
    0: failed to validate component output
    1: `task.return` requires the component model async feature (at offset 0x1b527)

I confirmed that I'm using the wasm-tools you provided here: cargo install --locked --git https://github.com/dicej/wasm-tools --branch async wasm-tools. Is there anything I missed?

@dicej
Copy link
Owner

dicej commented Jan 16, 2025

Hi @xdlin; thanks for reporting this. I've moved the contents of this repo into this PR as part of my effort to get everything upstreamed, so the code here (and the wasm-tools and wit-bindgen forks it points to) is no longer being maintained. Sorry for the confusion. I'll update the README.md to indicate the status.

Meanwhile, my wit-bindgen and wasm-tools forks have been merged upstream, so the latest releases of those crates have full async/future/stream support. However, part of that upstreaming work required adding a new, non-default wasmparser::WasmFeatures::COMPONENT_MODEL_ASYNC feature which is required to validate components which use any of the new async features. And since there's currently no way to specify extra features to wasm-tools component new, you have to do something like this instead:

wasm-tools component new --skip-validation module.wasm -o component.wasm
wasm-tools validate --features all component.wasm

@xdlin
Copy link
Author

xdlin commented Jan 16, 2025

Thank you for your quick response, I keep watching this PR as well :) I'm grateful for the info, I'll try with your PR later.

Previously I build my project which support Async Host <--> WASM <--> Async Guest with fp-bindgen, but it's inactive now, and I believe component mode with async is the future, so I'd like to switch to component mode once it's ready.

@xdlin
Copy link
Author

xdlin commented Jan 17, 2025

wasm-tools component new --skip-validation module.wasm -o component.wasm

I guess this is a necessary step for 'wasm-wasip1'?

sync WIT 1st try

I first tried with sync code, and with --target wasm32-wasip1, and run command:

wasm-tools component new --skip-validation ../target/wasm32-wasip1/debug/guest.wasm  -o component.wasm

I got this error:

error: failed to encode a component from module

Caused by:
    0: failed to decode world from module
    1: module was not valid
    2: failed to resolve import `wasi_snapshot_preview1::clock_time_get`
    3: module requires an import interface named `wasi_snapshot_preview1`

sync WIT with adapter

wasm-tools component new --skip-validation ../target/wasm32-wasip1/debug/guest.wasm  -o component.wasm --adapt ./wasi_snapshot_preview1.reactor.wasm 

After downloading the adapter and ran with this command, the sync code compiles

async WIT

When I change the WIT to async mode: generate!({async: true, ...}), I got this error:

error: failed to encode a component from module

Caused by:
    0: failed to decode world from module
    1: module was not valid
    2: failed to resolve import `[export]$root::[task-return]hello`
    3: module requires an import interface named `$root`

with target wasm32-wasip2

If I build with target wasm32-wasip2, I got a identical error like above:

...
= note: error: failed to encode component
          
          Caused by:
              0: failed to decode world from module
              1: module was not valid
              2: failed to resolve import `[export]$root::[task-return]hello`
              3: module requires an import interface named `$root`
          

error: could not compile `guest` (lib) due to 1 previous error

So the error happens with async code only, am I missed something?

my working env

  • rustc: 1.84.0 (9fc6b4312 2025-01-07)
  • wasm-tools: 1.223.0
  • wit-bindgen v0.37.0
  • wit-bindgen-rt v0.37.0

@dicej
Copy link
Owner

dicej commented Jan 17, 2025

Hmm, it seems to working for me. Here's what I did:

cargo install wasm-tools
mkdir -p foo/wit foo/guest/src foo/host/src
cd foo
cat >wit/round-trip.wit <<EOF
package local:local;

interface baz {
  foo: func(s: string) -> string;
}

world round-trip {
  import baz;
  export baz;
}
EOF
cat >guest/Cargo.toml <<EOF
[package]
name = "guest"
version = "0.1.0"
edition = "2021"

[dependencies]
futures = "0.3.30"
once_cell = "1.19.0"
wit-bindgen = "0.37.0"
wit-bindgen-rt = "0.37.0"

[lib]
crate-type = ["cdylib"]
EOF
cat >guest/src/lib.rs <<EOF
mod bindings {
    wit_bindgen::generate!({
        path: "../wit",
        world: "round-trip",
        async: true,
    });

    use super::Component;
    export!(Component);
}

use bindings::{exports::local::local::baz::Guest as Baz, local::local::baz};

struct Component;

impl Baz for Component {
    async fn foo(s: String) -> String {
        format!(
            "{} - exited guest",
            baz::foo(&format!("{s} - entered guest")).await
        )
    }
}
EOF
cargo build --manifest-path guest/Cargo.toml --release --target wasm32-wasip1
curl -OL https://github.com/bytecodealliance/wasmtime/releases/download/v28.0.1/wasi_snapshot_preview1.reactor.wasm
wasm-tools component new --adapt wasi_snapshot_preview1.reactor.wasm --skip-validation guest/target/wasm32-wasip1/release/guest.wasm -o guest.wasm
wasm-tools validate --features all guest.wasm
cat >host/Cargo.toml <<EOF
[package]
name = "host"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = { version = "1.0.81", features = ["backtrace"] }
tokio = { version = "1.36.0", features = ["fs", "macros", "rt-multi-thread", "time"] }
wasmtime = { git = "https://github.com/dicej/wasmtime", branch = "async", features = ["component-model-async"] }
wasmtime-wasi = { git = "https://github.com/dicej/wasmtime", branch = "async" }
EOF
cat >host/src/main.rs <<EOF
use {
    anyhow::Result,
    std::{env, future::Future, time::Duration},
    tokio::fs,
    wasmtime::{
        component::{self, Component, Linker, PromisesUnordered, ResourceTable},
        Config, Engine, Store, StoreContextMut,
    },
    wasmtime_wasi::{WasiCtx, WasiCtxBuilder, WasiView},
};

mod round_trip {
    wasmtime::component::bindgen!({
        trappable_imports: true,
        path: "../wit",
        world: "round-trip",
        concurrent_imports: true,
        concurrent_exports: true,
        async: true,
    });
}

struct Ctx {
    wasi: WasiCtx,
    table: ResourceTable,
}

impl WasiView for Ctx {
    fn table(&mut self) -> &mut ResourceTable {
        &mut self.table
    }
    fn ctx(&mut self) -> &mut WasiCtx {
        &mut self.wasi
    }
}

impl round_trip::local::local::baz::Host for Ctx {
    type Data = Ctx;

    #[allow(clippy::manual_async_fn)]
    fn foo(
        _: StoreContextMut<'_, Self>,
        s: String,
    ) -> impl Future<
        Output = impl FnOnce(StoreContextMut<'_, Self>) -> wasmtime::Result<String> + 'static,
    > + Send
           + 'static {
        async move {
            tokio::time::sleep(Duration::from_millis(10)).await;
            component::for_any(move |_: StoreContextMut<'_, Self>| {
                Ok(format!("{s} - entered host - exited host"))
            })
        }
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let mut config = Config::new();
    config.wasm_component_model(true);
    config.wasm_component_model_async(true);
    config.async_support(true);

    let engine = Engine::new(&config)?;

    let make_store = || {
        Store::new(
            &engine,
            Ctx {
                wasi: WasiCtxBuilder::new().inherit_stdio().build(),
                table: ResourceTable::default(),
            },
        )
    };

    let component = Component::new(&engine, &fs::read(env::args().nth(1).unwrap()).await?)?;

    let mut linker = Linker::new(&engine);

    wasmtime_wasi::add_to_linker_async(&mut linker)?;
    round_trip::RoundTrip::add_to_linker(&mut linker, |ctx| ctx)?;

    let mut store = make_store();

    let round_trip =
        round_trip::RoundTrip::instantiate_async(&mut store, &component, &linker).await?;

    // Start three concurrent calls and then join them all:
    let mut promises = PromisesUnordered::new();
    for _ in 0..3 {
        promises.push(
            round_trip
                .local_local_baz()
                .call_foo(&mut store, "hello, world!".to_owned())
                .await?,
        );
    }

    while let Some(value) = promises.next(&mut store).await? {
        println!("{value}");
    }

    Ok(())
}
EOF
cargo run --manifest-path host/Cargo.toml --release -- guest.wasm

Does the above work for you? If so, maybe you can modify that to suit your needs. Otherwise, please let me know where it fails on your end.

@xdlin
Copy link
Author

xdlin commented Jan 17, 2025

Wow, your version works~ And after comparing the code, I find the reason, I exported the function in world directly:

package local:local;

world round-trip {
  export foo: func(s: string) -> string;
}

And the related guest code:

mod bindings {
    wit_bindgen::generate!({
        path: "../wit",
        world: "round-trip",
        async: true,
    });

    use super::Component;
    export!(Component);
}

use bindings::{Guest as Baz};

struct Component;

impl Baz for Component {
    async fn foo(s: String) -> String {
        format!(
            "{} - exited guest",
            s
        )
    }
}

With this direct export function in world, I can reproduce the error

@xdlin
Copy link
Author

xdlin commented Jan 17, 2025

Thanks for your help and your great work on this feature, I can continue my experiment without any blockers after changing the WIT syntax a little bit

@dicej
Copy link
Owner

dicej commented Jan 17, 2025

Ah interesting; I know I tested direct world exports in WAT, but maybe I'm missing a test for that which uses wit-bindgen. I'll make sure to add one and fix any issues that pop up. Thanks for letting me know!

@dicej
Copy link
Owner

dicej commented Jan 17, 2025

FYI, here's the fix for world-level exports: bytecodealliance/wasm-tools#1979

@xdlin
Copy link
Author

xdlin commented Feb 7, 2025

hi @dicej when I tried to integrate this feature into my own project, it was OK before, but with recent change, I got thie compile error:

   Compiling wasmtime-environ v30.0.0 (https://github.com/dicej/wasmtime?branch=async#3d6cd202)
   Compiling cranelift-frontend v0.117.0 (https://github.com/dicej/wasmtime?branch=async#3d6cd202)
   Compiling cranelift-native v0.117.0 (https://github.com/dicej/wasmtime?branch=async#3d6cd202)
error[E0026]: variant `wasmparser::CanonicalFunction::TaskReturn` does not have a field named `result`
   --> /home/linxiangdong/.cargo/git/checkouts/wasmtime-a7bebf70e035e99d/3d6cd20/crates/environ/src/component/translate.rs:636:69
    |
636 |                         wasmparser::CanonicalFunction::TaskReturn { result } => {
    |                                                                     ^^^^^^
    |                                                                     |
    |                                                                     variant `wasmparser::CanonicalFunction::TaskReturn` does not have this field
    |                                                                     help: `wasmparser::CanonicalFunction::TaskReturn` has a field named `type_index`

error[E0027]: pattern does not mention field `type_index`
   --> /home/linxiangdong/.cargo/git/checkouts/wasmtime-a7bebf70e035e99d/3d6cd20/crates/environ/src/component/translate.rs:636:25
    |
636 |                         wasmparser::CanonicalFunction::TaskReturn { result } => {
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing field `type_index`
    |
help: include the missing field in the pattern
    |
636 |                         wasmparser::CanonicalFunction::TaskReturn { result, type_index } => {
    |                                                                           ~~~~~~~~~~~~~~
help: if you don't care about this missing field, you can explicitly ignore it
    |
636 |                         wasmparser::CanonicalFunction::TaskReturn { result, type_index: _ } => {
    |                                                                           ~~~~~~~~~~~~~~~~~
help: or always ignore missing fields here
    |
636 |                         wasmparser::CanonicalFunction::TaskReturn { result, .. } => {
    |                                                                           ~~~~~~

Some errors have detailed explanations: E0026, E0027.

I'm not sure whether this is the proper place to report this, if necessary, I can put this in another repo with full context.

@xdlin
Copy link
Author

xdlin commented Feb 7, 2025

It looks like the definition of TaskReturn was changed in wasmparer v0.225, but wasmtime async branch somehow still points to v0.224

@dicej
Copy link
Owner

dicej commented Feb 9, 2025

@xdlin we've shifted development over to the main branch of the https://github.com/bytecodealliance/wasip3-prototyping repository, so you should no longer use my branch. Sorry for the confusion. That branch should be compatible with the latest version of wasmparser.

Also, please keep in mind that all of this code is very experimental and won't be supported until it's been merged into the main branch of the bytecodealliance/wasmtime repo. Feel free to report bugs on the wasip3-prototyping repo as you run into them, but I can't guarantee we'll address them promptly. And note that more ABI-breaking changes are expected, so binaries built today won't necessarily run on future versions of wasip3-prototyping, and it won't always be obvious which versions/forks/branches of wasm-tools, wit-bindgen, and wasmtime are compatible with each other.

@xdlin
Copy link
Author

xdlin commented Feb 9, 2025

@dicej Thank you so much for your response, I'm totally understand that this feature is in early stage and not production ready yet. While since I'm so eager for this feature, and would like to integrate that into my own project as soon as possible, so I'm happy to take the ABI breaking risk, and hopefully I can provide some valuable feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants