Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple results comparison in benchmark scripts #14196

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Eason0729
Copy link
Contributor

Which issue does this PR close?

Close #13446

Rationale for this change

It would be useful to display the difference among multiple benchmark runs.

See issue.

What changes are included in this PR?

I modify bench.sh and compare.py to accept multiple arguments

Are these changes tested?

Are there any user-facing changes?

No.

@Eason0729
Copy link
Contributor Author

I spent some time trying to make it(compare.py) more readable, and I ended up with original code.
Let me know if there is a better way to do this.

Anyway, I thinks it ready for review.

@Eason0729 Eason0729 marked this pull request as ready for review January 22, 2025 02:53
@alamb
Copy link
Contributor

alamb commented Jan 23, 2025

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

@Eason0729
Copy link
Contributor Author

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

../bench.sh compare main main2 main3
Comparing main2 main3 to main
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query        ┃       main ┃      main2 ┃ Change on main2 ┃      main3 ┃ Change on main3 ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Qsort utf8   │ 25191.05ms │ 25191.05ms │       no change │ 25191.05ms │       no change │
│ Qsort int    │ 30908.16ms │ 30908.16ms │       no change │ 40908.16ms │    1.32x slower │
│ Qsort        │ 25652.09ms │ 25652.09ms │       no change │ 25652.09ms │       no change │
│ decimal      │            │            │                 │            │                 │
│ Qsort        │ 32701.67ms │ 32701.67ms │       no change │ 32701.67ms │       no change │
│ integer      │            │            │                 │            │                 │
│ tuple        │            │            │                 │            │                 │
│ Qsort utf8   │ 25588.63ms │ 25588.63ms │       no change │ 25588.63ms │       no change │
│ tuple        │            │            │                 │            │                 │
│ Qsort mixed  │ 29063.22ms │ 29063.22ms │       no change │ 29063.22ms │       no change │
│ tuple        │            │            │                 │            │                 │
└──────────────┴────────────┴────────────┴─────────────────┴────────────┴─────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary              ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (main)              │ 169104.82ms │
│ Average Time (main)            │  28184.14ms │
│ Total Time (main2)             │ 169104.82ms │
│ Average Time (main2)           │  28184.14ms │
│ Queries Faster (main2)         │           0 │
│ Queries Slower (main2)         │           0 │
│ Queries with No Change (main2) │           6 │
│ Total Time (main3)             │ 179104.82ms │
│ Average Time (main3)           │  29850.80ms │
│ Queries Faster (main3)         │           0 │
│ Queries Slower (main3)         │           1 │
│ Queries with No Change (main3) │           5 │
└────────────────────────────────┴─────────────┘

@alamb alamb mentioned this pull request Feb 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support multiple (>2) results comparison in benchmark scripts
2 participants