Reproducible builds

[xofyarg]

Summary

At version 0.17, the compiler is able to generate the same output with the same input. But after upgrading to 1.0.2, the compilation results differ each time even the wasm file is the same. I tried LLVM and cranelift. I'm using the default options, not sure if I missed anything.

[syrusakbary]

I believe this is due to the introduction of parallel compilation, at least on the LLVM side.
I've also read that latest version of cranelift may have issues with determinism (trying to search the source where I read it).

How important is deterministic compilation for your use case @xofyarg? Also, have you tried singlepass?

[xofyarg]

I just tried singlepass, same issue.

My use case is to pre-compile the binary in one location, and distribute it worldwide. The build service is stateless, so it doesn't know if the source is changed or not. The new version introduced some unnecessary pressure on the pipeline, but it acceptable at the moment.

[syrusakbary]

Thanks for the info!
Could it be possible to have a PR opened in this repo testing the deterministic behavior so we can work in a fix?

[xofyarg]

I add some tests to show the issue in PR #2184.

[nlewycky]

I've identified two problems. The easier one is that ModuleInfo::function_names is a HashMap. Maybe this is because it tends to be sparse, as not all functions have names? The iteration order of a hashmap is indeterminate so bincode's output for serializing one is indeterminate. We could either serialize this one differently (for example, in sorted order) or use a different type for it (for example, a BTreeMap).

The other issue is inside compiler-cranelift/src/compiler.rs, in a parallel loop over all function_body_inputs we have dwarf_frametable.lock().expect("").add_fde() which essentially appends bytes to the end of dwarf_frametable. This causes the FDEs in the frametable to be in an order that depends on which parallel compilation happens to finish first. We should defer this until after the parallel part and do them in function order (or, use a semaphore to block until earlier functions are done, that way we can still have compilation of later functions run in parallel with generation of unwind info for this function).

I haven't looked into compilers or engines besides cranelift+jit.

[syrusakbary]

I've investigated a bit more the issue. If you are using the JIT engine, the issue comes from Cranelift. It emits non-deterministic results when compiling functions.

Here are the results for Engines:

• JIT: deterministic
• Native: not deterministic

Compilers:

• Singlepass: deterministic
• Cranelift: non-deterministic
• LLVM: deterministic

[syrusakbary]

Note: what was not deterministic was the serialization due to use of HashMaps

[syrusakbary]

This PR is testing that we have deterministic builds:
#2184