Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GitHub Action workflow to run benchmarks on CI for pull requests #153

Merged
merged 1 commit into from
Feb 23, 2021

Conversation

giordano
Copy link
Member

No description provided.

@giordano
Copy link
Member Author

giordano commented Feb 22, 2021

It works: https://github.com/Team-RADDISH/ParticleDA.jl/blob/ceec818fcd6ea3b07c6f9e5abfb63d68fd0ab582/2021/02/22/164051/result.md 🥳 As I suspected, benchmark/benchmarks.jl should be already committed to master for this to work (if I understand correctly that's because it tries to compare performance with master, so the file needs to be there already).

@giordano
Copy link
Member Author

giordano commented Feb 23, 2021

Of the benchmark results published yesterday, only the first and the last seems meaningful (no performance change between the baseline on master and this PR), while the other two report a 50% speedup (with no changes to the code...).

I'm not sure what's going on, I've run the BenchmarkCI code locally multiple times and always found consistent results between the target and the baseline. I suspect that running benchmark on GitHub-hosted machines may not be the best idea possible (perhaps machine are randomly hit by other load?). In the past I tried to run benchmarks on Travis and results were unreliable at best, but "totally meaningless" is a better depiction of my experience, performance was extremely poor overall. Azure/GitHub machines are better than Travis, but may still be not always reliable.

My suggestion would be to merge this, but take results with a grain of salt.

@giordano giordano marked this pull request as ready for review February 23, 2021 13:00
@giordano
Copy link
Member Author

Not sure this explains the issue (especially, not the difference between target and baseline), but the two "good runs" were using Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, while the other two were on Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz.

@giordano giordano merged commit 61f33d7 into master Feb 23, 2021
@giordano giordano deleted the mg/benchmarks-ci branch February 23, 2021 17:56
@giordano giordano linked an issue Feb 26, 2021 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Benchmark suite
2 participants