Move wasm, wasi_snapshot_preview1, ieee754 and similar folders out of internal into utils or helpers

[skelouse]

Is your feature request related to a problem? Please describe.
Currently we are not able to decode a wasm module before loading it, exporting these was the only way to load it.

Describe the solution you'd like
Move wasm, wasi_snapshot_preview1, ieee754 and similar folders out of internal into utils or helpers

Describe alternatives you've considered
Forking the wazero repository, and exporting the internals ourselves.

[codefromthecrypt]

Hi, we have a internal package hierarchy defined intentionally, as described in RATIONALE.md

We don't currently plan to expose the raw wasm model as we have internal hooks we routinely add to that. To decode separate from instantiation, we have the Runtime.CompileModule and the current compile-time (decode time) customizations are available in CompileConfig.

Maybe you can mention specifically why you want to separate out decode from config independent of Runtime.CompileModule? Is there some customization you need to do?

[codefromthecrypt]

@skelouse fyi we will be spinning out a new repo, watzero for the text format, and that will need to have access to the binary types anyway. So, if the goal is manipulation not otherwise done in config, watch tetratelabs/wabin#1

Also, to elaborate compile includes decode, validation (terribly important) and in the case of the compiler backend, translating to machine level assembly, there are more things involved if the goal is caching. This should also land soon #618

We may also end up with a code builder here, but it won't be the same as the internal package for reasons including mutability which can subvert our validation, as well package cycles.

Anyway, I hope these breadcrumbs help, but also very curious about the use cases that motivated this issue. At the end of the month we'll start a chat channel (when we do our first beta), until then we can use this issue to elaborate. cheers!

[skelouse]

@codefromthecrypt

Runtime.CompileModule will fail when calling r.store.Engine.CompileModule as our dependencies are not known at that time. This is why we need to parse the module import section first, load dependencies then load the module.

[codefromthecrypt]

can you make a failing test for that please? CompileModule shouldn't rely on dependencies. in fact CompileModule exposes its imports exactly for this reason

[codefromthecrypt]

ex imports and exports were exposed recently so people could poke at which dependencies are needed prior to instantiation here:

wazero/config.go

Lines 263 to 290 in 9414b0b

	// CompiledModule is a WebAssembly 1.0 module ready to be instantiated (Runtime.InstantiateModule) as an api.Module.
	//
	// In WebAssembly terminology, this is a decoded, validated, and possibly also compiled module. wazero avoids using
	// the name "Module" for both before and after instantiation as the name conflation has caused confusion.
	// See https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#semantic-phases%E2%91%A0
	//
	// Note: Closing the wazero.Runtime closes any CompiledModule it compiled.
	type CompiledModule interface {
	// Name returns the module name encoded into the binary or empty if not.
	Name() string
	// ImportedFunctions returns all the imported functions
	// (api.FunctionDefinition) in this module or nil if there are none.
	//
	// Note: Unlike ExportedFunctions, there is no unique constraint on
	// imports.
	ImportedFunctions() []api.FunctionDefinition
	// ExportedFunctions returns all the exported functions
	// (api.FunctionDefinition) in this module keyed on export name.
	ExportedFunctions() map[string]api.FunctionDefinition
	// Close releases all the allocated resources for this CompiledModule.
	//
	// Note: It is safe to call Close while having outstanding calls from an
	// api.Module instantiated from this.
	Close(context.Context) error
	}

[codefromthecrypt]

@skelouse did any of the tips above work for you?

[codefromthecrypt]

I exposed the binary parsers in the new repo here, though it needs some polishing. Feel free to open an issue https://github.com/tetratelabs/watzero if you still need to manipulate wasm.

[codefromthecrypt]

@skelouse ok this should be more precisely what you were looking for https://github.com/tetratelabs/wabin

[skelouse]

@skelouse ok this should be more precisely what you were looking for https://github.com/tetratelabs/wabin

Thanks, that's what I needed! I'll get back to you when I find out more info.