Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make CI deterministic try 2 #4190

Merged
merged 10 commits into from
Oct 14, 2024
Merged

Make CI deterministic try 2 #4190

merged 10 commits into from
Oct 14, 2024

Conversation

RunDevelopment
Copy link
Contributor

Continuation of #4138.

@daxpedda daxpedda marked this pull request as draft October 12, 2024 20:36
@RunDevelopment
Copy link
Contributor Author

Yoooooo! The reference test is finally deterministic! It finally works!

I'm going to polish this PR tomorrow.

@RunDevelopment
Copy link
Contributor Author

RunDevelopment commented Oct 13, 2024

Alright, done. The output of JS code gen is finally deterministic.

The cause

The chain that caused the nondeterministic code gen is as follows:

  1. Import IDs are nondeterministic
  2. This causes the order in which import adapters are added to be nondeterministic
  3. This causes adapter IDs to be nondeterministic
  4. This causes the order in which adapters are iterated to be nondeterministic
  5. This makes code gen nondeterministic.

So what are import IDs? Import IDs are the arena ID from walrus' ModuleImports. So they are just an index into a memory arena.

Since these IDs are nondeterministic, I suspect that imports are added into the arena in a nondeterministic order. However, I have no idea why that happens. My guess is that walrus just adds them in whatever order they appear in the WASM binary. This would explain why the order seems to depend on compiler versions and OS, and why #4138 (where I made everything in wasm bindgen deterministic) didn't work.

Honestly, I'm not sure what the bug here is. Is the bug that import IDs are nondeterministic, or that wasm bindgen relied on them being deterministic?

The fix

The fix I went with is quite simple: sort imports by name and not ID. Since the non-determinism of import IDs "infects" adapter IDs, I made sure that code gen doesn't depend on their order by sorting imports (and adapters of imports) by import name.

This works, but isn't ideal IMO. I would have preferred to make adapter IDs deterministic, but I couldn't. The wit code handling imports and import IDs is too complex for me.

@RunDevelopment RunDevelopment marked this pull request as ready for review October 13, 2024 12:05
Copy link
Collaborator

@daxpedda daxpedda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Thank you for digging into this!

For future archeologists clarity:

wasm-bindgens ouput is already deterministic, just not across different versions of wasm-bindgen, Rust and the OS.
This doesn't fix any bug and has no user-facing effects apart of the ordering of imported functions in the JS shim.

In the end, this is just to make integration tests in wasm-bindgen less annoying by not having to re-order imports all the time.

crates/cli-support/src/js/mod.rs Show resolved Hide resolved
@daxpedda daxpedda added the waiting for author Waiting for author to respond label Oct 13, 2024
@daxpedda daxpedda merged commit ae1d105 into rustwasm:main Oct 14, 2024
41 checks passed
@daxpedda
Copy link
Collaborator

Again, thank you for figuring this out!
This makes long-term maintenance a bit less annoying, so its very appreciated!

@RunDevelopment RunDevelopment deleted the det-ci-2 branch October 14, 2024 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting for author Waiting for author to respond
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants