Add host env via trait with proc macro

[MarkMcCaskey]

This PR includes a trait-based host-env init and deinit flow (including the plumbing to call these functions) as well as a procedural macro to generate this code.

For example,

/// The environment provided to the WASI imports.
#[derive(Debug, Clone, WasmerEnv)]
pub struct WasiEnv {
    state: Arc<Mutex<WasiState>>,
    #[wasmer(export)]
    memory: LazyInit<Memory>,
    #[wasmer(export(name = "real_name"))]
    sum: LazyInit<NativeFunc<(i32, i32), i32>>,
}

will generate code roughly looking like:

impl ::wasmer::WasmerEnv for WasiEnv {
    fn finish(&mut self, instance: &::wasmer::Instance) {
        let memory: Memory = instance.exports.get_with_generics("memory").unwrap();
        self.memory.initialize(memory);
        let sum: NativeFunc<(i32, i32), i32> = instance.exports.get_with_generics("real_name").unwrap();
        self.sum.initialize(sum);
    }
}
impl WasiEnv {
    pub fn memory_ref(&self) -> Option<&Memory> {
        self.memory.get()
    }
    pub fn sum_ref(&self) -> Option<&NativeFunc<(i32, i32), i32>> {
        self.sum.get()
    }
    pub unsafe fn memory_ref_unchecked(&self) -> &Memory {
        self.memory.get_unchecked()
    }
    pub unsafe fn sum_ref_unchecked(&self) -> &NativeFunc<(i32, i32), i32> {
        self.sum.get_unchecked()
    }
}

The exact details of the interface are subject to change and should be reviewed thoroughly, for example we will want to be able to handle errors properly instead of unwraping and assuming they don't happen.

note to self: The generated helper functions can be used in an unsound way if called before we instantiate, but it'd be nice to not have to use unsafe or deal with Option when being called from a host function...

Review

• Review lazy init abstraction
  • Review its thread-safety
  • Review its interface
• Review the WasmerEnv trait
  • Review the interface
  • Review the plumbing (function pointer casting, data flow, requirement that all types implement WasmerEnv)
• Review the proc macro
  • Review the implementation, likely simpler ways to do certain things but proc macros are kind of confusing
  • Review test coverage
  • Review test coverage of unhappy paths / error messages, etc
• Add a short description of the the change to the CHANGELOG.md file

[nlewycky]

Could you break the vmctx -> union part to its own PR?

[Hywan]

Why InitAfterInstance must implement Send? Is it to preserve the behavior of T in this case?

[MarkMcCaskey]

This was also for WasiEnv but more largely because the way Wasmer works, we can pass functions to other threads and call it meaning that it's Send... perhaps it can also be used in a Sync way but I didn't think so...

Someone filed an issue for us to review this and I agree, that we should probably audit what we actually allow users to do

[Hywan]

I don't understand what it does exactly, and why the name function_ptr? Can you explain please?

[MarkMcCaskey]

This is WasmerEnv::finish for the env used by this function, due to our heirarchy of types, we can't be explicit about the type signature because this is in vm and it takes &api::Instance as an argument.

It's really not ideal but I haven't come up with a better way to get access to the initializer function yet. This is the thing I was hoping to get the most feedback on early

[syrusakbary]

I'm unclear on why we need this here. The env finisher should live in the InstanceHandle, right?

[MarkMcCaskey]

Good question!

This is how the function pointers get into the InstanceHandle: we lose type information, but wasmer::externals::function::Function methods have access to the initializer, we have to store it somewhere for later and this is the best place I've found.

The little code window above that Ivan commented on is showing out of date code, but I left a comment there asking for feedback if anyone knows a better way to get this information to where we need it.

[syrusakbary]

Why don't store it in InstanceHandle ?

[MarkMcCaskey]

We do, but we have to get that information to the InstanceHandle.

The data flow starts when a user makes a new Function:

1. Function::new_with_env : by way of the trait constraint, we can get the initializer and turn it into a function pointer. We then put it in ExportFunction because it seemed like the most obvious place to put it.

2. We need this data when instantiating Instance::new, Module::instantiate , we only get a module and a resolver (the resolver is where all the functions are living)

3. We end up in engine::Artifact::instantiate here we extract these values from the Imports during instantiation and make it part of the InstanceHandle

4. Later we call a function on InstanceHandle to do something with this stored data

[syrusakbary]

The PR I think it moves to the good direction.

Feedback:

1. Test the wasm-c-api wasm_func_new_with_env with a custom finalizer
2. Test the behavior of a creating a custom environment. That the finalizer and free methods get called.

[syrusakbary]

How we can change the API?

[MarkMcCaskey]

There's no good way to do this that I can think of yet. The only way to avoid the transmutes would be to to make the WasmerEnv::init_with_instance function take void* directly... but that seems like a bad trade off.

[syrusakbary]

I'm not super sure why this is needed. Can you provide some context @MarkMcCaskey ?

[MarkMcCaskey]

Which part? The function_ptr at all or the transmute?

We need the function pointer to initialize the data, so far this is the most direct place to put it so we can access where we need it.

We have to transmute it because WasmerEnv isn't known to the vm so we can't use the real name of the type, I have some work in progress trying to make it so vm can see WasmerEnv but it's a bit stuck

[syrusakbary]

We should move this out of this struct.

[MarkMcCaskey]

This is the same as #1739 (comment) ; it may be possible but I'm not exactly sure how yet

[syrusakbary]

I really think this doesn't belong here.

[syrusakbary]

I actually like more UnsafeMutableLegacyEnv. Verbose but it conveys the meaning. MutableLegacyEnv is also good

[MarkMcCaskey]

So this change is actually fixing an issue from the find and replace done on in another PR. It should really be split out and merged to master separately, but I accidentally renamed a struct with the same name in another place by doing bulk find and replace, this just corrects that mistake

[MarkMcCaskey]

bors r+

[bors]

Build succeeded:

• Test on windows
• Test cross-compile on macos
• Audit
• Test on linux
• Test on linux-aarch64
• Code lint
• Test on macos
• Test cross-compile on linux