Reverts

[michaelsbradleyjr]

Maybe there should be a PR template that populates the description box with a commented-out section like

<!--
💡 Benchmark results in the project's readme are gathered from builds/runs peformed in a Microsoft Azure free tier VM:

Standard F4s v2 (4 vcpus, 8 GiB memory)

Your changes or new implementation may perform better in a different environment, e.g. your local machine or a different cloud provider.

To avoid having your contributions reverted because of performance regressions, consider doing a build/run in an equivalent Azure VM before submitting your PR. 🙏
-->

I'm suggesting it because reverts for perf regressions keep landing in main and contributors may appreciate a reminder to try out their changes in an equivalent environment.

For example, I was disappointed to learn a consistent 50% speedup on my local machines turns into a 200%+ slowdown in the benchmarking VM.

I now have my own Standard F4s v2 (4 vcpus, 8 GiB memory) VM and am hunting down the problem, but I might have worked with one from the start if it had been suggested in the process of submitting a PR.

[jinyus]

The onus is on me to test on the VM before merging. I will now do that before merging PRs.
The VM costs $123/month so I don't think it's feasible to ask contributors to run tests on them.

[michaelsbradleyjr]

A contributor can spin up and tear down a free tier VM w/in the free-period of signup. Most contributors will probably be done with the process in a few days, so wouldn't cost them anything.

Still, yeah, it's not a great ask, but some kind of heads-up when submitting a PR would be a good idea, I think, even if the final "consider doing..." sentence is left out.

[cyrusmsk]

Maybe @microsoft will fund to this project a small Azure machine :) 2 of their languages are battle-tested here :)

[michaelsbradleyjr]

That would be cool, not sure if it will happen, though.

If it does, it would also be cool if there's a bot/action that upon request (via PR comment) does 5..20..60k runs for the new/changed implementation/s and then posts the results in a PR comment. I've seen conceptually similar actions implemented in other projects, takes a bit of work but isn't rocket science.

For examples nim-lang/Nim has a custom bisect action that can be run on request in an issue/PR comment: nim-lang/Nim#22826 (comment).

See:

• https://github.com/nim-lang/Nim/blob/devel/.github/workflows/bisects.yml
• https://github.com/juancarlospaco/nimrun-action

So for this project would be a comment like:

!run swift swift_con

Could be restricted to use by jinyus if it was getting abused.

[jinyus]

Sounds good but results can vary with github actions.

I made a small userscript where I can click a button to copy the docker command needed to run the benchmark on a PR branch so all I have to do is paste it in the VM and run it.

[michaelsbradleyjr]

Sounds good but results can vary with github actions.

Yep, I had in mind it triggering a build/run on dedicated VM not GHA.

I made a small userscript where I can click a button to copy the docker command needed to run the benchmark on a PR branch so all I have to do is paste it in the VM and run it.

Nice!