-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add custom CSV and Markdown reporters for benchmark results #109
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the output has is for machine use anyway, IMO we should just export individual samples and leave all the statistics to the postprocessing script (e.g. scipy + matplotlib). That would be strictly more general.
15c323d
to
4d66f96
Compare
I've changed the CSV reporter to also include the raw samples. I've also introduced a new Markdown reporter that can be used to produce human-readable results alongside the CSV. Simply run with |
Thanks! Most of this looks great, however for the Markdown output, we technically need to escape all the special characters so test / benchmark names aren't interpreted as formatting. For example, the CDAG benchmarks have an underscore as in This is less about nitpicking and more towards the question whether we actually need proper Markdown output, or if a Edit: I just realized GitHub has a (searchable) CSV renderer, which is kind of neat, but it unfortunately doesn't right-align numbers. |
It looks like |
As discussed during yesterday's meeting, there is value in GitHub's Markdown diff, so we'll keep it. I've added rudimentary escaping. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The markdown report in ci/perf/gpuc1_bench.md
has not been updated since escaping was added. Otherwise LGTM!
- Include raw samples in output - Don't print part number for distinguishing generator values, instead detect uses of GENERATE and print a warning.
This generates markdown benchmark reports intended for human consumption. Run benchmarks with --reporter celerity-benchmark-csv::results.csv \ --reporter celerity-benchmark-md::results.md to generate both CSV and Markdown reports simultaneously.
55a176a
to
7c5e82a
Compare
I've manually added the escaping to the relevant benchmark names; rebased onto master. |
This adds a custom CSV reporter for producing machine-readable benchmarking results. You can see an example output in the updated results for gpuc1 (I noticed some of the numbers diverge quite wildly from the previous results, not sure how reliable those are - cc @PeterTh).
To use the reporter, simply call the benchmarks with
--reporter celerity-benchmark-csv
.