Add wasm2c output samples to tests

[keithw]

Per #1877 (comment), adding some samples of wasm2c output to the repository so we can track them when it changes. The three samples are:

• a minimal module
• a module that exports a single add function
• a WASI "Hello, world." command, along with a mock WASI implementation of fd_write and proc_exit that it links with

[sbc100]

Wow, this is kind of cool.

I was thinking of simpler test that just runs wasm2c and includes the output without pigging backing on the spec test stuff, and without even trying to actually compile the code, let alone run it, but this more compressive that what I had in mind.

I'm not sure about the name of the sub-directory? Do we need to sub-directory at all? I'm also not sure about how to select things for inclusion here.

I think these three all seem reasonable. We might want to add another one later which tries to use every possible feature of wasm2c (so any wasm2c change would cause its output to change).

Do you think there is any value in the simpler testing that just shows the output of wasm2c (without compiling and running it as if it was a spec test)?

[keithw]

Wow, this is kind of cool.

I was thinking of simpler test that just runs wasm2c and includes the output without pigging backing on the spec test stuff, and without even trying to actually compile the code, let alone run it, but this more compressive that what I had in mind.

I'm not sure about the name of the sub-directory? Do we need to sub-directory at all?

Not at all -- here's a commit that moves them into test/wasm2c and gets rid of the outputs subdirectory.

I'm also not sure about how to select things for inclusion here.

I think these three all seem reasonable. We might want to add another one later which tries to use every possible feature of wasm2c (so any wasm2c change would cause its output to change).

Agreed -- this does seem like a good idea.

Do you think there is any value in the simpler testing that just shows the output of wasm2c (without compiling and running it as if it was a spec test)?

I do... and we can certainly add a --no-compile argument to these tests if you think preferable. It did seem that the simplest way to get here (from where we are) was this small change to add the --print-output arg to run-spec-wasm2c.py, and then compiling and running sort of comes for free. I guess I was less eager to make intrusive changes to run-tests.py to add a new "tool", and possibly a new script to print out the wasm2c-generated output since I'm not sure we can shell out to "cat" on Windows, etc. :-/

[keithw]

@sbc100 @shravanrn Could you render a verdict on whether this is still wanted? It could be nice with the changes to c-writer.cc and the source templates that are in the hopper, or we could decide it's not worth it... I'm happy either way but wanted to bump this and hopefully move towards consensus one way or the other.

[shravanrn]

Sorry, I am just trying to get caught up on this discussion. Is the question do we want a simple hello-world with wasi example?
If so, yeah, I think that would be great!

I currently have been an uncommitted test which I manually run
See here (Feel free to borrow any code if it looks useful)

So this definitely looks great to me :)

(Let me know if you wanted me to code review)

[sbc100]

I think the idea is great but we might want to tweak the implementation a little. I'm happy to land this as is an do followups though.

1. I'm not sure about using run-spec-wasm2c for this. I seems to be overloading this script a little. Would just running wasm2c directly be better/enough?
2. I wonder if it would make diffs and reading easier if we use separate output files rather than inlining all the generated code. For example, doign it this way makes harder to distinguish header changes from source changes.

[sbc100]

Is there any reason to include the register and assert_return line, if we are just interested in storing the output code? i.e. Isn't the functionality of adding already covered by the spec tests?

[keithw]

I had initially done the register so that run-spec-wasm2c.py would give the module a nicer name (e.g. adder_instance_t instead of add_0_wasm_instance_t) but now that we're using wasm2c directly this doesn't matter -- removed.

[keithw]

@shravanrn Awesome, would be grateful for your input. I think the backstory here was that if we're making a lot of changes to c-writer.cc and the wasm2c source templates, it can be challenging to figure out "how does this affect the wasm2c output?" just by looking at the diffs on GitHub. So @sbc100 had suggested maybe we should commit some fully-worked examples in the repo that have an input .wat file and the corresponding expected .h and .c outputs, and now every time we change the wasm2c output, the PR will have to rebase those tests to match the new output and that diff will be readable on GitHub. The pro is that the bottom-line impact to wasm2c output will be more easily visible when doing reviews on GitHub; the con is that it's more friction on each change.

[sbc100]

@shravanrn Awesome, would be grateful for your input. I think the backstory here was that if we're making a lot of changes to c-writer.cc and the wasm2c source templates, it can be challenging to figure out "how does this affect the wasm2c output?" just by looking at the diffs on GitHub. So @sbc100 had suggested maybe we should commit some fully-worked examples in the repo that have an input .wat file and the corresponding expected .h and .c outputs, and now every time we change the wasm2c output, the PR will have to rebase those tests to match the new output and that diff will be readable on GitHub. The pro is that the bottom-line impact to wasm2c output will be more easily visible when doing reviews on GitHub; the con is that it's more friction on each change.

I think that given how easy it is to run ./test/run-tests.py test/wasm2c/* -r, the cost is pretty minimal.

[keithw]

I think the idea is great but we might want to tweak the implementation a little. I'm happy to land this as is an do followups though.

1. I'm not sure about using run-spec-wasm2c for this.  I seems to be overloading this script a little.  Would just running wasm2c directly be better/enough?

I just gave it a shot and it went a lot smoother than I thought it would when I first did it the other way. Done (and run-spec-wasm2c.py is now unchanged).

2. I wonder if it would make diffs and reading easier if we use separate output files rather than inlining all the generated code.   For example, doign it this way makes harder to distinguish header changes from source changes.

That also seems doable but a slightly bigger lift from where we are now... Would you want to have like add_c.txt and add_h.txt and each one runs wasm2c and then prints out just the .c or just the .h file?

[shravanrn]

Separate output files if possible would be nice, but happy to go with whatever is easily possible.

Other than that the only one thing I wanted to add --- Not sure if this already exists and I missed it, but we possibly want to document somewhere how we can update the tests in case we change c-writer.cc

[sbc100]

Yes, perhaps even in the test failure message it could say something like If these changes are expected you can re-run with --rebase to update the expectations.. then you wouldn't even need to reach for the docs.