Replaced traverse benchmarks with something more representative
#4403
Conversation
| case h :: t => G.map2Eval(f(h), Eval.defer(loop(t)))(_ :: _) | ||
| case Nil => Eval.now(G.pure(Nil)) | ||
| if (i == length * 0.3) { | ||
| throw Failure |
There was a problem hiding this comment.
I'm surprised to see us using throw in these benchmarks along with Eval. Can you add a comment since it is non-idiomatic to do so?
There was a problem hiding this comment.
Eval secretly supports short-circuiting, which means it's a halfway-decent approximate for IO. Ultimately, I would be willing to bet that 90% of traverse usage in the wild has IO as the base applicative. In this case, I was just trying to caliper down on the short-circuiting scenario without adding extra overhead that would muddy the constant factors.
The other advantage to doing it this way is it allows us to compare like-for-like. If we write the same traverse process by hand (which I have now done locally in Cats Effect), we can compare directly and see the overhead of traverse itself, allowing us to explore different strategies.
| // the unit of CPU work per iteration | ||
| private[this] val Work: Long = 10 |
There was a problem hiding this comment.
In terms of JMH, we are going to burn CPU time per invocation here. Isn't it?
There was a problem hiding this comment.
Yep! Specifically, per iteration of the traverse itself.
| .value | ||
| } | ||
|
|
||
| bh.consume(result.value) |
There was a problem hiding this comment.
Just wondering, is consuming of result value a thing here?
There was a problem hiding this comment.
In this case, it's almost certainly unnecessary, but in general this is a good thing to do since it prevents the combination of inlining and escape analysis from inadvertently eliding the code path you actually intended to measure, resulting in over-fitting.
I'll plop some results in here tonight. I don't want to base on my regular machine because benchmarking using Apple Silicon is… very very deceptive. Once I'm back home I'll use another home desktop to run the tests. |
|
Basically replacing all of the
traversebenchmarks with something more representative and projecting it onto a more useful scale. This makes it possible to directly compareTraverse[List].traverseandList#map, for example, as well as ultimately experiment with different implementation strategies that potentially avoid extraEvalwrapping.