Streaming PromQL engine benchmark tooling

[charleskorn]

What this PR does

This PR adds a tool to capture peak memory utilisation of the PromQL engine benchmarks added in #7693.

It also refactors the benchmarks to use a Mimir ingester rather than a TSDB directly. This makes it possible to run the benchmarks with the ingester out-of-process. This is more representative of how the query engine will be used in the real world.

The benchmark-query-engine tool takes advantage of this to run the ingester in its process, and run the benchmark in a separate process, ensuring that the peak memory utilisation recorded by benchmark-query-engine does not include memory used by the ingester. It captures the peak RSS of the benchmark process and reports this as the peak memory utilisation of the benchmark. There's more details about benchmark-query-engine in the readme.

It produces output like the following, allowing it to be used with tools like benchstat, as well as the engine benchmark comparison script:

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
BenchmarkQuery/a_1,_instant_query/streaming-10         	   10228	    115452 ns/op	   17640 B/op	     302 allocs/op     65372160 B
BenchmarkQuery/a_1,_instant_query/standard-10         	    8766	    122042 ns/op	   20730 B/op	     362 allocs/op     66224128 B

Full benchmark results

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
                                                                 │      standard      │              streaming              │
                                                                 │       sec/op       │    sec/op     vs base               │
Query/a_1,_instant_query                                                 123.9µ ± 14%   115.3µ ±  2%   -6.93% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                       125.9µ ±  8%   113.4µ ±  1%   -9.94% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                    130.4µ ±  2%   118.6µ ±  2%   -9.02% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                   187.6µ ±  2%   178.7µ ±  3%   -4.76% (p=0.002 n=6)
Query/a_100,_instant_query                                               612.6µ ±  4%   572.2µ ±  1%   -6.61% (p=0.002 n=6)
Query/a_100,_range_query_with_1_step                                     608.4µ ±  4%   577.6µ ±  4%   -5.06% (p=0.009 n=6)
Query/a_100,_range_query_with_100_steps                                  1.093m ±  3%   1.028m ±  1%   -5.93% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                 5.573m ±  4%   5.091m ±  1%   -8.65% (p=0.002 n=6)
Query/a_2000,_instant_query                                              7.665m ±  5%   7.178m ±  3%   -6.35% (p=0.009 n=6)
Query/a_2000,_range_query_with_1_step                                    7.883m ±  7%   7.300m ±  1%   -7.39% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                 15.89m ±  7%   15.06m ±  1%   -5.23% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                97.36m ±  6%   88.61m ±  1%   -8.99% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                       128.4µ ±  3%   120.5µ ± 12%        ~ (p=0.132 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                             132.2µ ±  5%   119.0µ ±  7%   -9.99% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                          139.8µ ±  4%   124.7µ ±  9%  -10.84% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                         256.9µ ± 10%   185.1µ ±  2%  -27.93% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                     594.7µ ±  2%   568.5µ ±  3%   -4.41% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                           602.8µ ±  4%   568.9µ ±  1%   -5.62% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                        1.630m ±  3%   1.175m ± 11%  -27.91% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                      10.325m ±  1%   5.856m ±  1%  -43.28% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                    7.321m ±  4%   6.915m ±  1%   -5.55% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                          7.455m ±  3%   7.576m ±  5%        ~ (p=0.699 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                       26.51m ±  2%   17.76m ±  0%  -33.01% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                      194.3m ±  6%   104.4m ±  1%  -46.25% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                       1190.5µ ± 11%   752.1µ ±  0%  -36.83% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                      96.20m ±  1%   53.19m ±  1%  -44.71% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                      1.974 ±  1%    1.067 ±  6%  -45.93% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                       771.6µ ± 15%   543.0µ ±  7%  -29.62% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                             776.5µ ±  5%   540.7µ ±  1%  -30.37% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                         1342.2µ ±  1%   980.7µ ±  1%  -26.94% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                         6.174m ±  2%   4.774m ±  1%  -22.68% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                     43.27m ±  0%   33.47m ±  1%  -22.64% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                           43.87m ±  3%   33.98m ±  1%  -22.56% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                        97.46m ±  2%   75.80m ±  8%  -22.22% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                       582.3m ±  1%   456.3m ±  2%  -21.64% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                    807.7m ±  1%   618.6m ±  1%  -23.41% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                          819.2m ±  2%   626.1m ±  0%  -23.58% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                        1.912 ±  1%    1.477 ±  0%  -22.77% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                      11.606 ±  1%    9.155 ±  2%  -21.12% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                            124.2µ ± 19%   115.6µ ±  3%   -6.89% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                  127.2µ ±  3%   115.7µ ±  1%   -9.04% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                               159.4µ ±  3%   123.4µ ±  1%  -22.59% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                              454.9µ ±  3%   182.2µ ± 11%  -59.95% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                          595.1µ ±  1%   569.2µ ±  3%   -4.36% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                                605.2µ ±  2%   570.8µ ±  1%   -5.69% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                             1.225m ±  1%   1.027m ±  1%  -16.17% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                            6.825m ±  1%   5.231m ±  0%  -23.35% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                         7.422m ±  2%   7.149m ±  1%   -3.68% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1_step                               7.644m ±  1%   7.199m ±  1%   -5.82% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                            18.88m ±  1%   15.82m ±  1%  -16.21% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                          154.65m ± 11%   92.52m ±  9%  -40.17% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                     161.2µ ± 22%   153.6µ ± 11%        ~ (p=0.132 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                           159.5µ ±  8%   146.2µ ± 15%   -8.36% (p=0.026 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                        233.6µ ±  2%   177.5µ ±  4%  -24.00% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                       879.4µ ±  1%   486.3µ ±  3%  -44.70% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                   2.652m ±  0%   2.536m ±  4%   -4.37% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                         2.733m ±  1%   2.589m ±  1%   -5.28% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                      9.096m ±  1%   5.140m ±  1%  -43.49% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                     67.28m ±  1%   29.10m ±  1%  -56.75% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                  47.19m ±  1%   46.02m ±  3%   -2.46% (p=0.041 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                        49.58m ±  0%   46.65m ±  1%   -5.92% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                    187.43m ±  1%   94.84m ±  1%  -49.40% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                   1565.5m ±  2%   557.1m ±  4%  -64.42% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                    162.4µ ±  5%   151.5µ ±  4%   -6.73% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                          171.7µ ±  8%   149.9µ ± 21%  -12.66% (p=0.041 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                       362.6µ ±  6%   192.2µ ±  6%  -47.00% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                     1950.2µ ±  3%   540.0µ ± 15%  -72.31% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                  2.618m ±  3%   2.499m ±  9%        ~ (p=0.065 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                        2.745m ±  9%   2.520m ±  2%   -8.22% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                     6.919m ± 21%   5.337m ±  1%  -22.86% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                    43.58m ± 11%   30.59m ±  7%  -29.82% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                 46.85m ±  1%   45.35m ±  5%        ~ (p=0.240 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                       48.35m ± 11%   44.88m ±  8%   -7.19% (p=0.026 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                   126.73m ±  7%   97.80m ±  6%  -22.83% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                  1018.1m ± 10%   559.1m ±  0%  -45.09% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                          161.9µ ±  2%   148.0µ ±  1%   -8.58% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step                165.7µ ±  6%   159.8µ ±  4%   -3.56% (p=0.004 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps             345.5µ ±  2%   193.4µ ±  3%  -44.04% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps           1817.7µ ±  1%   552.1µ ±  9%  -69.63% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                        2.518m ±  3%   2.387m ±  1%   -5.20% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step              2.649m ±  3%   2.439m ±  5%   -7.92% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps           9.422m ±  4%   5.921m ±  2%  -37.15% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps          70.57m ±  1%   33.97m ±  0%  -51.86% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                       44.42m ±  2%   42.93m ±  1%   -3.35% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step             46.59m ±  1%   43.10m ±  1%   -7.50% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps          189.8m ±  1%   112.5m ±  1%  -40.72% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps        1609.8m ±  0%   655.5m ±  1%  -59.28% (p=0.002 n=6)
geomean                                                                  5.722m         4.336m        -24.22%

                                                                 │      standard      │               streaming                │
                                                                 │        B/op        │      B/op       vs base                │
Query/a_1,_instant_query                                                20.24Ki ±  0%    17.25Ki ±  0%   -14.76% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                      20.54Ki ±  0%    17.26Ki ±  0%   -15.98% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                   21.21Ki ±  1%    17.96Ki ±  0%   -15.28% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                  27.36Ki ±  0%    25.87Ki ±  2%    -5.46% (p=0.002 n=6)
Query/a_100,_instant_query                                              140.7Ki ±  0%    122.3Ki ±  0%   -13.11% (p=0.002 n=6)
Query/a_100,_range_query_with_1_step                                    137.2Ki ±  0%    123.1Ki ±  0%   -10.22% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                 196.8Ki ±  0%    182.5Ki ±  0%    -7.29% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                773.4Ki ±  0%    761.7Ki ±  1%    -1.52% (p=0.002 n=6)
Query/a_2000,_instant_query                                             2.423Mi ±  0%    2.336Mi ±  1%    -3.62% (p=0.002 n=6)
Query/a_2000,_range_query_with_1_step                                   2.344Mi ±  0%    2.514Mi ±  1%    +7.24% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                3.434Mi ±  0%    3.350Mi ±  1%    -2.45% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                               14.51Mi ±  0%    14.44Mi ±  1%    -0.48% (p=0.041 n=6)
Query/rate(a_1[1m]),_instant_query                                      22.42Ki ±  0%    17.65Ki ±  0%   -21.25% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                            22.85Ki ±  0%    17.66Ki ±  0%   -22.70% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                         23.40Ki ±  0%    18.39Ki ±  0%   -21.42% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                        29.30Ki ±  1%    26.14Ki ±  2%   -10.77% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                    165.2Ki ±  0%    125.2Ki ±  0%   -24.22% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                          161.6Ki ±  0%    126.1Ki ±  0%   -21.99% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                       221.6Ki ±  0%    185.6Ki ±  0%   -16.21% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                      762.4Ki ±  0%    724.8Ki ±  1%    -4.93% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                   2.871Mi ±  0%    2.403Mi ±  1%   -16.28% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                         2.793Mi ±  0%    2.567Mi ±  3%    -8.08% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                      3.878Mi ±  0%    3.408Mi ±  2%   -12.14% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                     14.33Mi ±  0%    13.91Mi ±  1%    -2.88% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                       84.54Ki ± 14%    91.79Ki ± 14%    +8.58% (p=0.004 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                     5.191Mi ±  1%    5.038Mi ±  5%         ~ (p=0.310 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                    1.043Gi ±  0%    3.088Gi ±  0%  +195.99% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                     665.90Ki ±  3%    63.47Ki ±  2%   -90.47% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                           666.50Ki ±  2%    63.51Ki ±  1%   -90.47% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                        688.60Ki ±  5%    63.30Ki ±  2%   -90.81% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                       681.83Ki ±  4%    68.42Ki ±  5%   -89.97% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                    5.204Mi ±  2%    4.475Mi ±  1%   -14.01% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                          5.220Mi ±  1%    4.504Mi ±  0%   -13.71% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                       5.293Mi ±  3%    4.500Mi ±  1%   -14.98% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                      5.756Mi ±  5%    4.801Mi ±  6%   -16.60% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                   87.73Mi ±  1%    86.23Mi ±  1%    -1.71% (p=0.004 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                         87.64Mi ±  1%    87.09Mi ±  1%    -0.63% (p=0.026 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                      92.23Mi ±  1%   119.18Mi ±  1%   +29.23% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                     126.9Mi ±  1%    406.9Mi ±  0%  +220.58% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                           22.00Ki ±  0%    17.86Ki ±  0%   -18.81% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                 22.51Ki ±  0%    17.99Ki ±  0%   -20.11% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                              39.48Ki ±  0%    19.70Ki ±  0%   -50.10% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                            195.02Ki ±  0%    35.83Ki ±  1%   -81.63% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                         170.8Ki ±  0%    123.9Ki ±  0%   -27.47% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                               170.9Ki ±  0%    124.2Ki ±  0%   -27.33% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                            247.7Ki ±  0%    185.8Ki ±  0%   -25.00% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                           972.8Ki ±  0%    818.6Ki ±  1%   -15.85% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                        3.027Mi ±  0%    2.167Mi ±  1%   -28.42% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1_step                              3.069Mi ±  0%    2.168Mi ±  0%   -29.34% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                           4.174Mi ±  0%    3.287Mi ±  0%   -21.25% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                          15.46Mi ±  0%    14.85Mi ±  1%    -3.91% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                    28.97Ki ±  0%    23.28Ki ±  0%   -19.64% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                          29.43Ki ±  0%    23.39Ki ±  0%   -20.52% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                       51.40Ki ±  0%    27.90Ki ±  0%   -45.72% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                     258.93Ki ±  0%    75.70Ki ±  2%   -70.76% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                 1059.9Ki ±  0%    751.9Ki ±  0%   -29.06% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                       1095.9Ki ±  0%    768.2Ki ±  0%   -29.90% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                     3.505Mi ±  0%    1.176Mi ±  0%   -66.44% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                   25.922Mi ±  2%    5.343Mi ±  0%   -79.39% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                 21.30Mi ±  0%    17.46Mi ±  1%   -18.04% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                       22.17Mi ±  1%    18.25Mi ±  1%   -17.68% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                    76.84Mi ±  1%    25.72Mi ±  2%   -66.53% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                   798.4Mi ±  0%    108.7Mi ±  1%   -86.39% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                   30.26Ki ±  0%    23.43Ki ±  0%   -22.57% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                         31.32Ki ±  0%    24.18Ki ±  0%   -22.80% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                     130.55Ki ±  0%    33.51Ki ±  0%   -74.33% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                    1043.0Ki ±  0%    120.5Ki ±  1%   -88.45% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                1024.3Ki ±  0%    737.0Ki ±  0%   -28.05% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                      1046.9Ki ±  0%    738.9Ki ±  0%   -29.42% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                    1.445Mi ±  0%    1.063Mi ±  0%   -26.46% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                   5.665Mi ±  0%    4.594Mi ±  1%   -18.90% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                20.42Mi ±  0%    16.70Mi ±  1%   -18.23% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                      20.87Mi ±  1%    16.63Mi ±  1%   -20.30% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                   27.35Mi ±  0%    23.64Mi ±  0%   -13.57% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                 283.83Mi ±  0%    92.51Mi ±  0%   -67.41% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                         33.55Ki ±  0%    23.85Ki ±  0%   -28.89% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step               34.77Ki ±  0%    24.58Ki ±  0%   -29.30% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps           119.78Ki ±  0%    33.92Ki ±  0%   -71.68% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps           901.1Ki ±  0%    117.6Ki ±  1%   -86.95% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                      1160.5Ki ±  0%    756.1Ki ±  0%   -34.85% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step            1186.5Ki ±  0%    757.2Ki ±  0%   -36.19% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps          1.575Mi ±  0%    1.085Mi ±  0%   -31.13% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps         5.560Mi ±  0%    4.422Mi ±  1%   -20.47% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                      23.14Mi ±  0%    17.02Mi ±  1%   -26.45% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step            23.77Mi ±  0%    17.08Mi ±  1%   -28.13% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps         30.27Mi ±  3%    24.02Mi ±  0%   -20.63% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps       282.43Mi ±  0%    88.92Mi ±  1%   -68.51% (p=0.002 n=6)
geomean                                                                 1.194Mi          782.4Ki         -36.01%

                                                                 │      standard      │             streaming              │
                                                                 │     allocs/op      │  allocs/op   vs base               │
Query/a_1,_instant_query                                                   362.0 ± 0%    302.0 ± 0%  -16.57% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                         368.0 ± 0%    302.0 ± 0%  -17.93% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                      373.0 ± 0%    307.0 ± 0%  -17.69% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                     403.0 ± 0%    337.0 ± 0%  -16.38% (p=0.002 n=6)
Query/a_100,_instant_query                                                1.978k ± 0%   1.910k ± 0%   -3.44% (p=0.002 n=6)
Query/a_100,_range_query_with_1_step                                      1.984k ± 0%   1.911k ± 0%   -3.68% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                   2.489k ± 0%   2.417k ± 0%   -2.89% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                  5.504k ± 0%   5.427k ± 0%   -1.40% (p=0.002 n=6)
Query/a_2000,_instant_query                                               32.80k ± 0%   32.70k ± 0%   -0.29% (p=0.002 n=6)
Query/a_2000,_range_query_with_1_step                                     32.80k ± 0%   32.73k ± 0%   -0.22% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                  42.84k ± 0%   42.74k ± 0%   -0.24% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                 102.9k ± 0%   102.8k ± 0%   -0.11% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                         405.0 ± 0%    312.0 ± 0%  -22.96% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                               411.0 ± 0%    312.0 ± 0%  -24.09% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                            416.0 ± 0%    317.0 ± 0%  -23.80% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                           443.0 ± 0%    344.0 ± 0%  -22.35% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                      2.426k ± 0%   2.019k ± 0%  -16.78% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                            2.431k ± 0%   2.021k ± 0%  -16.87% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                         2.937k ± 0%   2.526k ± 0%  -13.99% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                        5.654k ± 0%   5.236k ± 0%   -7.40% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                     40.86k ± 0%   34.72k ± 0%  -15.04% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                           40.86k ± 0%   34.74k ± 0%  -14.99% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                        50.90k ± 0%   44.75k ± 0%  -12.08% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                      105.01k ± 0%   98.85k ± 0%   -5.87% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                          672.5 ± 0%    573.0 ± 0%  -14.80% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                       28.64k ± 0%   28.20k ± 0%   -1.52% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                      576.2k ± 0%   564.0k ± 0%   -2.12% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                         651.0 ± 0%    540.5 ± 0%  -16.97% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                               657.0 ± 0%    540.5 ± 0%  -17.73% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                            660.0 ± 0%    544.0 ± 0%  -17.58% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                           680.5 ± 0%    566.0 ± 0%  -16.83% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                      25.37k ± 0%   24.92k ± 0%   -1.77% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                            25.38k ± 0%   24.93k ± 0%   -1.74% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                         25.68k ± 0%   25.24k ± 0%   -1.70% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                        27.83k ± 0%   27.36k ± 0%   -1.69% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                     501.8k ± 0%   491.9k ± 0%   -1.98% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                           501.9k ± 0%   495.7k ± 0%   -1.24% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                        509.9k ± 0%   503.7k ± 0%   -1.21% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                       552.1k ± 0%   546.0k ± 0%   -1.09% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                              400.0 ± 0%    311.0 ± 0%  -22.25% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                    409.0 ± 0%    315.0 ± 0%  -22.98% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                 810.0 ± 0%    320.0 ± 0%  -60.49% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                               4442.0 ± 0%    350.5 ± 0%  -92.11% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                           2.115k ± 0%   1.919k ± 0%   -9.27% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                                 2.028k ± 0%   1.924k ± 0%   -5.13% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                              2.930k ± 0%   2.429k ± 0%  -17.08% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                             9.546k ± 0%   5.446k ± 0%  -42.95% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                          34.81k ± 0%   32.72k ± 0%   -6.01% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1_step                                32.86k ± 0%   32.72k ± 0%   -0.41% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                             43.29k ± 0%   42.76k ± 0%   -1.22% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                            107.0k ± 0%   102.9k ± 0%   -3.80% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                       503.0 ± 0%    403.0 ± 0%  -19.88% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                             508.0 ± 0%    407.0 ± 0%  -19.88% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                         1033.0 ± 0%    438.0 ± 0%  -57.60% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                        5715.0 ± 0%    618.0 ± 0%  -89.19% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                    11.57k ± 0%   10.76k ± 0%   -7.02% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                          11.20k ± 0%   11.18k ± 0%   -0.18% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                       35.85k ± 0%   14.19k ± 0%  -60.42% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                     250.77k ± 0%   32.24k ± 0%  -87.15% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                   224.5k ± 0%   211.4k ± 0%   -5.81% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                         229.3k ± 3%   222.6k ± 0%   -2.90% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                      689.7k ± 0%   277.2k ± 0%  -59.81% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                    4736.4k ± 0%   639.4k ± 0%  -86.50% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                      516.0 ± 0%    408.0 ± 0%  -20.93% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                            534.0 ± 0%    433.0 ± 0%  -18.91% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                        2348.0 ± 0%    463.0 ± 0%  -80.28% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                      18737.0 ± 0%    644.0 ± 0%  -96.56% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                   11.36k ± 0%   10.65k ± 0%   -6.19% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                         10.80k ± 0%   10.68k ± 0%   -1.10% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                      15.59k ± 0%   13.69k ± 0%  -12.18% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                     49.84k ± 0%   31.75k ± 0%  -36.28% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                  219.0k ± 0%   206.9k ± 0%   -5.52% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                        208.7k ± 2%   206.9k ± 0%   -0.87% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                     269.0k ± 0%   267.2k ± 0%   -0.66% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                    671.5k ± 0%   627.8k ± 0%   -6.50% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                            572.0 ± 0%    417.0 ± 0%  -27.10% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step                  588.0 ± 0%    442.0 ± 0%  -24.83% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps              1804.0 ± 0%    472.0 ± 0%  -73.84% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps            12774.5 ± 0%    635.0 ± 0%  -95.03% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                         13.81k ± 0%   11.26k ± 0%  -18.41% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step               13.25k ± 0%   11.29k ± 0%  -14.82% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps            18.04k ± 0%   14.30k ± 0%  -20.69% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps           50.49k ± 0%   30.57k ± 0%  -39.46% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                        267.1k ± 0%   218.9k ± 0%  -18.06% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step              275.5k ± 1%   218.9k ± 0%  -20.55% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps           317.5k ± 0%   279.2k ± 0%  -12.07% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps          683.7k ± 0%   603.8k ± 0%  -11.68% (p=0.002 n=6)
geomean                                                                   11.51k        7.999k       -30.51%

                                                                 │      standard      │               streaming                │
                                                                 │         B          │       B         vs base                │
Query/a_1,_instant_query                                                 63.12Mi ± 1%    62.38Mi ±  2%         ~ (p=0.193 n=6)
Query/a_1,_range_query_with_1_step                                       63.34Mi ± 1%    62.35Mi ±  1%    -1.57% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                    62.98Mi ± 1%    62.30Mi ±  1%         ~ (p=0.074 n=6)
Query/a_1,_range_query_with_1000_steps                                   62.11Mi ± 1%    62.94Mi ±  1%    +1.33% (p=0.024 n=6)
Query/a_100,_instant_query                                               58.24Mi ± 1%    57.31Mi ±  2%    -1.60% (p=0.026 n=6)
Query/a_100,_range_query_with_1_step                                     58.59Mi ± 2%    57.47Mi ±  1%    -1.91% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                  58.81Mi ± 1%    60.73Mi ±  2%    +3.25% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                 60.54Mi ± 1%    77.14Mi ±  1%   +27.42% (p=0.002 n=6)
Query/a_2000,_instant_query                                              59.90Mi ± 1%    62.32Mi ±  2%    +4.04% (p=0.002 n=6)
Query/a_2000,_range_query_with_1_step                                    60.21Mi ± 1%    66.04Mi ±  2%    +9.68% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                 67.19Mi ± 2%   131.01Mi ± 10%   +94.99% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                127.0Mi ± 2%    294.2Mi ±  2%  +131.65% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                       62.39Mi ± 1%    62.09Mi ±  1%         ~ (p=0.132 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                             62.50Mi ± 1%    62.28Mi ±  1%         ~ (p=0.240 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                          62.58Mi ± 1%    61.97Mi ±  1%         ~ (p=0.102 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                         61.20Mi ± 2%    63.40Mi ±  1%    +3.59% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                     58.14Mi ± 1%    57.45Mi ±  1%    -1.18% (p=0.041 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                           58.59Mi ± 1%    57.97Mi ±  2%    -1.05% (p=0.009 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                        58.22Mi ± 1%    60.88Mi ±  1%    +4.56% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                       59.54Mi ± 1%    77.36Mi ±  2%   +29.93% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                    60.82Mi ± 2%    62.60Mi ±  1%    +2.93% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                          60.45Mi ± 1%    66.93Mi ±  2%   +10.71% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                       65.98Mi ± 2%   128.84Mi ±  1%   +95.26% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                      124.9Mi ± 1%    245.9Mi ±  2%   +96.97% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                        60.66Mi ± 1%    79.76Mi ±  7%   +31.49% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                      95.04Mi ± 3%   171.91Mi ±  8%   +80.88% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                     571.8Mi ± 6%    411.9Mi ±  1%   -27.97% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                       59.80Mi ± 0%    60.22Mi ±  1%         ~ (p=0.087 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                             59.86Mi ± 1%    60.30Mi ±  1%    +0.73% (p=0.026 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                          59.07Mi ± 0%    59.60Mi ±  1%    +0.90% (p=0.015 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                         57.20Mi ± 1%    57.27Mi ±  2%         ~ (p=0.937 n=6)
Query/rate(a_100[1d]),_instant_query                                     62.77Mi ± 1%    62.41Mi ±  1%         ~ (p=0.818 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                           63.30Mi ± 2%    61.91Mi ±  2%    -2.18% (p=0.015 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                        62.60Mi ± 4%    62.95Mi ±  2%         ~ (p=0.589 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                       57.02Mi ± 4%    66.06Mi ±  2%   +15.87% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                    69.81Mi ± 4%    70.06Mi ±  4%         ~ (p=0.937 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                          70.14Mi ± 2%    71.41Mi ±  4%         ~ (p=0.123 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                       70.43Mi ± 1%   103.52Mi ±  4%   +46.98% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                      102.5Mi ± 2%    149.6Mi ±  4%   +46.03% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                            62.55Mi ± 1%    62.03Mi ±  1%         ~ (p=0.255 n=6)
Query/sum(a_1),_range_query_with_1_step                                  62.77Mi ± 2%    62.05Mi ±  2%         ~ (p=0.132 n=6)
Query/sum(a_1),_range_query_with_100_steps                               60.30Mi ± 2%    63.07Mi ±  1%    +4.60% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                              58.60Mi ± 1%    68.80Mi ±  1%   +17.40% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                          58.13Mi ± 2%    57.42Mi ±  1%    -1.22% (p=0.011 n=6)
Query/sum(a_100),_range_query_with_1_step                                59.18Mi ± 1%    57.78Mi ±  1%    -2.36% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                             58.12Mi ± 1%    57.97Mi ±  1%         ~ (p=0.193 n=6)
Query/sum(a_100),_range_query_with_1000_steps                            60.63Mi ± 1%    58.84Mi ±  1%    -2.96% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                         59.71Mi ± 1%    59.85Mi ±  0%         ~ (p=0.619 n=6)
Query/sum(a_2000),_range_query_with_1_step                               60.48Mi ± 1%    59.14Mi ±  1%    -2.22% (p=0.004 n=6)
Query/sum(a_2000),_range_query_with_100_steps                            67.00Mi ± 2%    59.27Mi ±  1%   -11.54% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                          124.11Mi ± 1%    63.62Mi ±  1%   -48.73% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                     60.84Mi ± 1%    60.27Mi ±  1%    -0.95% (p=0.037 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                           61.57Mi ± 1%    60.92Mi ±  2%    -1.05% (p=0.026 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                        60.21Mi ± 2%    60.57Mi ±  1%         ~ (p=0.180 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                       58.71Mi ± 1%    62.34Mi ±  1%    +6.19% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                   58.16Mi ± 1%    58.12Mi ±  1%         ~ (p=1.000 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                         58.49Mi ± 1%    58.43Mi ±  1%         ~ (p=0.699 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                      60.84Mi ± 1%    63.12Mi ±  1%    +3.75% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                     80.94Mi ± 1%    89.30Mi ±  3%   +10.34% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                  68.64Mi ± 2%    67.55Mi ±  2%    -1.58% (p=0.041 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                        69.71Mi ± 3%    68.90Mi ±  4%         ~ (p=0.310 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                     121.4Mi ± 1%    142.8Mi ±  1%   +17.63% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                    510.8Mi ± 3%    333.1Mi ±  1%   -34.80% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                    60.80Mi ± 1%    60.54Mi ±  2%         ~ (p=0.485 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                          61.00Mi ± 1%    61.01Mi ±  1%         ~ (p=0.420 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                       59.25Mi ± 1%    63.79Mi ±  1%    +7.66% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                      58.75Mi ± 1%    75.98Mi ±  2%   +29.34% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                  58.17Mi ± 1%    58.35Mi ±  2%         ~ (p=0.485 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                        59.19Mi ± 1%    57.73Mi ±  1%    -2.47% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                     61.34Mi ± 1%    59.04Mi ±  2%    -3.74% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                    78.71Mi ± 1%    62.90Mi ±  2%   -20.09% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                 67.48Mi ± 1%    66.99Mi ±  1%         ~ (p=0.058 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                       68.99Mi ± 2%    66.40Mi ±  2%    -3.76% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                   108.41Mi ± 6%    65.85Mi ±  2%   -39.25% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                  269.94Mi ± 3%    67.16Mi ±  2%   -75.12% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                          60.66Mi ± 1%    60.49Mi ±  1%         ~ (p=0.589 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step                60.85Mi ± 1%    60.70Mi ±  1%         ~ (p=1.000 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps             58.93Mi ± 1%    63.50Mi ±  1%    +7.76% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps            58.41Mi ± 1%    76.84Mi ±  1%   +31.54% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                        58.36Mi ± 1%    58.38Mi ±  1%         ~ (p=0.732 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step              58.73Mi ± 1%    58.39Mi ±  1%         ~ (p=0.240 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps           60.38Mi ± 1%    58.86Mi ±  1%    -2.51% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps          76.89Mi ± 3%    61.48Mi ±  1%   -20.05% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                       67.27Mi ± 1%    67.18Mi ±  1%         ~ (p=0.961 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step             68.03Mi ± 2%    67.16Mi ±  1%    -1.29% (p=0.041 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps         104.61Mi ± 4%    65.08Mi ±  2%   -37.79% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps        264.35Mi ± 2%    67.02Mi ±  1%   -74.65% (p=0.002 n=6)
geomean                                                                  71.25Mi         71.77Mi          +0.73%

I've also modified the number of histogram buckets created for each test series to make benchmark setup faster.

Notes to reviewers:

• I don't love how much low-level plumbing was required to start the ingester and distributor queryable. Very open to other ideas that would make this simpler, while still allowing for benchmarks and tests to be run with go test ... without any other preparation (eg. building binaries or Docker images)
• I suspect there's more I could do to make the behaviour and performance of ingesters more consistent and predictable during the benchmark - very keen to hear feedback on this
• Similarly, I suspect there's more I could do to make the benchmark setup time faster - it currently takes ~1m on my machine to load all the test data into the ingester on each run. Again, very keen to hear ideas on how to improve this (eg. if there are settings on the ingester that can be tweaked for this).
  • Another option would be to have a way to generate the data ahead of time and reuse it for multiple benchmark runs, assuming the test data has not changed.

Which issue(s) this PR fixes or relates to

(none)

Checklist

• [n/a] Tests updated.
• [n/a] Documentation added.
• [n/a] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• [n/a] about-versioning.md updated with experimental features.

[pracucci]

Nice work! I agree on the value of running the benchmarks against ingester and Mimir (distributor) queryable. We get a benchmark closer to the actual Mimir utilization and will help to spot if we're missing anything (e.g. we expect an optimization to trigger, but we don't see the impact on benchmark).

That being said, I'm not a big fan of tools/benchmark-query-engine/ because it partially re-implements go test. However, I don't know how to reliably capture the memory peak utilization in a different way (I thought about profiling, but I think it will not work for our purpose and, even if so, may be even more complicated that you did). All in all, I think it's good enough and we should move on ;)