Refactor and publish benchmarks

[travishathaway]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

Summary

This project has not been touched for some time; therefore, the goal of this epic is to revisit the work done here, possibly add to or remove some benchmarks, and get it ready for publishing elsewhere (e.g. conda.org of the conda documentation).

The work will be organized into two separate groups:

• Writing/editing of benchmarks
• Publishing results; this includes setting up GitHub actions to allow us to automatically update these numbers

Linked Issues & PRs

• Revise current benchmarks and see what still works/make sense and what should be removed
• Consider a way that we can write benchmarks with the projects themselves and have the framework be centralized
• Write new benchmarks as needed (based on research from the first item)
• Assess benchmark stability on CI runners (see Refactor and publish benchmarks #30 (comment) below)
• Figure out a method to integrate the usage ASV on individual pull requests
• Figure out a method to publish reports that can either be embedded to our documentation website or conda.org

[jaimergp]

This would be lovely but first we need to assess how stable the benchmarks are or, more accurately, how they are run.

If the plan is to run it on public CI, the variance of the runtime conditions makes it, essentially, a very expensive false positive generator :D This is the blog post I wrote about it. This other one is also useful to get an idea on how noisy it can get.

CI is only noisy due to run time though ¹, so if we can measure performance via other metrics (e.g. number of operations), then we should be ok. This blog post explains how cachegrind can be used to do so, but it assumes the benchmarks are merely CPU bound. We need to check how IO is accounted for in the tests.

Footnotes

1. GHA runners may use different CPUs:

   Dv2-series run on the 3rd Generation Intel® Xeon® Platinum 8370C (Ice Lake), Intel® Xeon® Platinum 8272CL (Cascade Lake), Intel® Xeon® 8171M 2.1GHz (Skylake), Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell), or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processors with the Intel Turbo Boost Technology 2.0.

   ↩