First iteration of streaming PromQL engine

[charleskorn]

What this PR does

This PR adds the first iteration of a streaming PromQL engine.

It supports a very limited feature set, and is far from production-ready.

It supports the following PromQL features:

• instant vector selectors (eg. my_metric{env="prod"} and my_metric{env="prod"} @ 123)
• sum aggregations with no grouping and with by (eg. sum(my_metric) or sum by (env) (my_metric))
• rate (eg. rate(my_metric[5m]))
• combinations of sum and rate (eg. sum by (env) (rate(my_metric[5m])))

A number of features will be added in subsequent PRs, including support for falling back to Prometheus' PromQL engine if an unsupported PromQL feature is used, and support for native histograms.

Before reviewing this PR, I'd recommend reading this short introduction to the engine I've included in this PR, as it gives an overview of how the engine is structured and how the various parts interact.

There are many improvements I'd like to make beyond what's in this PR, but I believe what is here is good enough to merge, and we can build upon what's here in future PRs.

Benchmark results

These results compare Prometheus' PromQL engine (standard below) with the streaming engine added in this PR (streaming below).

Note that while these benchmarks capture latency and memory allocations, they do not capture peak memory utilisation. Reducing peak memory utilisation is one of the main goals behind this engine, and capturing this in benchmarks will be added in a subsequent PR. Initial peak memory utilisation benchmark results can be seen in this presentation.

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/querier/engine/streaming
                                                                 │ /tmp/standard.8oda │        /tmp/streaming.45b6         │
                                                                 │       sec/op       │   sec/op     vs base               │
Query/a_1,_instant_query                                                 6.325µ ±  1%   3.481µ ± 2%  -44.96% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                       6.891µ ±  4%   3.513µ ± 1%  -49.02% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                   10.128µ ±  1%   6.580µ ± 1%  -35.04% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                   34.17µ ±  1%   27.97µ ± 1%  -18.14% (p=0.002 n=6)
Query/a_100,_instant_query                                               173.9µ ±  1%   175.0µ ± 1%        ~ (p=0.394 n=6)
Query/a_100,_range_query_with_1_step                                     185.9µ ±  1%   177.9µ ± 1%   -4.29% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                  514.1µ ±  1%   480.4µ ± 0%   -6.56% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                 2.830m ±  0%   2.598m ± 2%   -8.21% (p=0.002 n=6)
Query/a_2000,_instant_query                                              4.682m ± 16%   4.815m ± 8%        ~ (p=0.394 n=6)
Query/a_2000,_range_query_with_1_step                                    4.972m ±  6%   4.835m ± 2%        ~ (p=0.065 n=6)
Query/a_2000,_range_query_with_100_steps                                 12.93m ± 10%   11.94m ± 3%   -7.66% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                60.54m ±  1%   55.93m ± 2%   -7.63% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                       8.949µ ±  1%   4.196µ ± 1%  -53.11% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                             9.543µ ±  1%   4.247µ ± 1%  -55.49% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                         17.607µ ±  1%   8.123µ ± 0%  -53.87% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                         79.57µ ±  0%   36.86µ ± 1%  -53.67% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                     203.6µ ±  1%   159.9µ ± 0%  -21.46% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                           217.1µ ±  1%   166.3µ ± 0%  -23.40% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                       1022.8µ ±  1%   542.4µ ± 1%  -46.97% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                       7.365m ±  1%   3.362m ± 1%  -54.36% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                    5.015m ±  3%   4.587m ± 4%   -8.53% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                          5.493m ±  3%   4.645m ± 2%  -15.45% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                       23.44m ±  2%   13.48m ± 4%  -42.49% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                     152.85m ±  1%   71.73m ± 3%  -53.07% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                        768.8µ ±  0%   385.6µ ± 1%  -49.84% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                      79.36m ±  1%   37.83m ± 1%  -52.33% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                    1572.1m ±  0%   752.0m ± 1%  -52.17% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                       305.1µ ±  2%   159.2µ ± 1%  -47.81% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                             310.6µ ±  1%   163.5µ ± 1%  -47.38% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                          841.2µ ±  4%   591.0µ ± 3%  -29.74% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                         5.747m ±  1%   4.423m ± 0%  -23.03% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                     25.56m ±  1%   15.61m ± 1%  -38.92% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                           25.80m ±  0%   15.76m ± 1%  -38.93% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                        79.15m ±  1%   58.31m ± 1%  -26.33% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                       566.0m ±  1%   438.5m ± 1%  -22.52% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                    506.9m ±  0%   313.9m ± 2%  -38.07% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                          515.3m ±  0%   320.9m ± 1%  -37.73% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                        1.614 ±  2%    1.163 ± 2%  -27.95% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                      11.409 ±  1%    8.776 ± 1%  -23.08% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                            8.226µ ±  1%   4.369µ ± 1%  -46.89% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                  9.110µ ±  2%   4.373µ ± 0%  -52.00% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                              36.021µ ±  1%   7.705µ ± 1%  -78.61% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                             267.29µ ±  1%   31.23µ ± 1%  -88.32% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                          178.3µ ±  2%   174.4µ ± 1%   -2.14% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                                182.4µ ±  1%   176.7µ ± 1%   -3.11% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                             660.1µ ±  1%   480.9µ ± 0%  -27.15% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                            4.368m ±  1%   2.615m ± 1%  -40.15% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                         4.368m ±  8%   4.544m ± 3%        ~ (p=0.065 n=6)
Query/sum(a_2000),_range_query_with_1_step                               4.461m ±  2%   4.629m ± 2%   +3.78% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                            14.67m ±  1%   11.24m ± 3%  -23.37% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                          122.62m ±  0%   55.57m ± 1%  -54.68% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                     27.05µ ±  1%   21.35µ ± 0%  -21.07% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                           28.48µ ±  3%   21.66µ ± 2%  -23.93% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                       107.08µ ±  1%   55.74µ ± 0%  -47.95% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                       740.8µ ±  2%   299.8µ ± 1%  -59.53% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                   1.989m ±  1%   2.177m ± 4%   +9.47% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                         2.080m ±  3%   2.206m ± 3%   +6.03% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                     10.579m ±  2%   5.697m ± 2%  -46.15% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                     81.26m ±  1%   30.03m ± 0%  -63.05% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                  48.12m ±  1%   55.35m ± 2%  +15.01% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                        50.61m ±  1%   55.80m ± 1%  +10.26% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                     277.5m ±  1%   132.8m ± 1%  -52.15% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                   2017.0m ±  5%   617.4m ± 1%  -69.39% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                    30.30µ ±  1%   24.76µ ± 2%  -18.28% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                          34.68µ ±  1%   25.11µ ± 1%  -27.58% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                      342.40µ ±  1%   60.97µ ± 0%  -82.19% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                     3032.7µ ±  1%   318.0µ ± 0%  -89.51% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                  2.011m ±  2%   2.128m ± 3%   +5.83% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                        2.012m ±  4%   2.138m ± 3%   +6.26% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                     7.851m ±  3%   5.710m ± 2%  -27.27% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                    63.86m ±  1%   30.04m ± 2%  -52.97% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                 47.67m ±  1%   54.14m ± 0%  +13.58% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                       48.70m ±  3%   54.98m ± 1%  +12.89% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                    202.7m ±  1%   132.0m ± 1%  -34.88% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                  1447.9m ±  3%   608.8m ± 1%  -57.95% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                          34.55µ ±  1%   22.98µ ± 1%  -33.48% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step                39.31µ ±  1%   23.72µ ± 1%  -39.65% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps            355.74µ ±  0%   67.58µ ± 2%  -81.00% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps           3133.4µ ±  1%   403.6µ ± 3%  -87.12% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                        2.276m ±  2%   1.985m ± 2%  -12.80% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step              2.374m ±  2%   1.995m ± 3%  -15.95% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps          13.887m ±  3%   6.512m ± 3%  -53.11% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps         113.15m ±  1%   38.62m ± 0%  -65.87% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                       54.14m ±  1%   50.88m ± 2%   -6.02% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step             56.46m ±  1%   51.19m ± 1%   -9.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps          312.2m ±  1%   144.5m ± 1%  -53.70% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps        2485.8m ±  8%   778.0m ± 1%  -68.70% (p=0.002 n=6)
geomean                                                                  3.122m         1.893m       -39.36%

                                                                 │ /tmp/standard.8oda │          /tmp/streaming.45b6          │
                                                                 │        B/op        │      B/op       vs base               │
Query/a_1,_instant_query                                               5.470Ki ±   0%   2.758Ki ±   0%  -49.58% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                     5.806Ki ±   0%   2.758Ki ±   0%  -52.50% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                  6.229Ki ±   0%   3.181Ki ±   0%  -48.93% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                10.809Ki ±   0%   7.771Ki ±   0%  -28.10% (p=0.002 n=6)
Query/a_100,_instant_query                                             58.11Ki ±   0%   39.27Ki ±   0%  -32.42% (p=0.002 n=6)
Query/a_100,_range_query_with_1_step                                   53.80Ki ±   0%   39.28Ki ±   0%  -26.99% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                83.58Ki ±   0%   69.12Ki ±   0%  -17.30% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                               355.8Ki ±   0%   341.7Ki ±   0%   -3.96% (p=0.002 n=6)
Query/a_2000,_instant_query                                           1065.4Ki ±   0%   792.2Ki ±   1%  -25.65% (p=0.002 n=6)
Query/a_2000,_range_query_with_1_step                                  969.5Ki ±   0%   789.1Ki ±   1%  -18.61% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                               1.527Mi ±   0%   1.357Mi ±   1%  -11.09% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                              6.823Mi ±   0%   6.688Mi ±   1%   -1.97% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                     7.696Ki ±   0%   3.180Ki ±   0%  -58.69% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                           8.033Ki ±   0%   3.180Ki ±   0%  -60.42% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                        8.455Ki ±   0%   3.603Ki ±   0%  -57.39% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                      12.724Ki ±   0%   7.881Ki ±   0%  -38.06% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                   82.06Ki ±   0%   42.79Ki ±   0%  -47.86% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                         77.69Ki ±   0%   42.80Ki ±   0%  -44.91% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                     107.53Ki ±   0%   72.63Ki ±   0%  -32.45% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                     348.7Ki ±   0%   313.7Ki ±   0%  -10.05% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                 1523.1Ki ±   0%   854.5Ki ±   1%  -43.89% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                       1426.4Ki ±   0%   854.6Ki ±   1%  -40.08% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                     1.976Mi ±   0%   1.421Mi ±   1%  -28.09% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                    6.678Mi ±   0%   6.158Mi ±   1%   -7.79% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                      66.08Ki ±   0%   61.39Ki ±   1%   -7.09% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                    2.940Mi ±   0%   2.920Mi ±   2%        ~ (p=0.065 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                   57.49Mi ±   1%   57.34Mi ±   1%        ~ (p=0.394 n=6)
Query/rate(a_1[1d]),_instant_query                                    555.18Ki ±   0%   35.69Ki ±   0%  -93.57% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                          555.62Ki ±   0%   35.67Ki ±   0%  -93.58% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                       553.66Ki ±   0%   35.96Ki ±   0%  -93.50% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                      566.29Ki ±   0%   48.47Ki ±   0%  -91.44% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                   2.909Mi ±   0%   2.365Mi ±   0%  -18.67% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                         2.904Mi ±   0%   2.364Mi ±   0%  -18.62% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                      2.953Mi ±   1%   2.396Mi ±   0%  -18.88% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                     3.385Mi ±   7%   2.669Mi ±   1%  -21.15% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                  48.44Mi ±   1%   47.20Mi ±   0%   -2.56% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                        48.35Mi ±   0%   47.29Mi ±   1%   -2.18% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                     49.08Mi ±   0%   48.65Mi ±  64%        ~ (p=0.065 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                    53.51Mi ±   1%   52.03Mi ± 603%        ~ (p=0.065 n=6)
Query/sum(a_1),_instant_query                                          7.274Ki ±   0%   3.328Ki ±   0%  -54.25% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                7.774Ki ±   0%   3.328Ki ±   0%  -57.19% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                            25.989Ki ±   0%   3.751Ki ±   0%  -85.57% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                          192.313Ki ±   0%   8.347Ki ±   0%  -95.66% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                        91.31Ki ±   0%   40.71Ki ±   0%  -55.41% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                              90.28Ki ±   0%   40.71Ki ±   0%  -54.90% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                          137.89Ki ±   0%   70.56Ki ±   0%  -48.83% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                          572.0Ki ±   0%   343.2Ki ±   0%  -40.00% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                      1815.3Ki ±   0%   803.7Ki ±   0%  -55.72% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1_step                            1787.4Ki ±   0%   802.3Ki ±   0%  -55.11% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                          2.345Mi ±   0%   1.366Mi ±   0%  -41.75% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                         7.808Mi ±   0%   6.666Mi ±   0%  -14.63% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                  16.307Ki ±   0%   7.202Ki ±   0%  -55.83% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                        16.683Ki ±   0%   7.202Ki ±   0%  -56.83% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                      40.96Ki ±   0%   10.60Ki ±   0%  -74.13% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                    262.46Ki ±   0%   42.28Ki ±   0%  -83.89% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                1066.7Ki ±   0%   489.8Ki ±   0%  -54.08% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                      1068.8Ki ±   0%   489.9Ki ±   0%  -54.17% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                   3686.6Ki ±   0%   818.0Ki ±   0%  -77.81% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                  26.827Mi ±   0%   3.727Mi ±   0%  -86.11% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                20.11Mi ±   0%   10.14Mi ±   3%  -49.59% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                      20.50Mi ±   0%   10.18Mi ±   3%  -50.34% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                   78.34Mi ±   0%   16.84Mi ±   4%  -78.51% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                 599.45Mi ±  57%   76.82Mi ±   3%  -87.19% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                 19.881Ki ±   0%   7.892Ki ±   0%  -60.30% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                       21.713Ki ±   0%   7.893Ki ±   0%  -63.65% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                    252.68Ki ±   0%   11.29Ki ±   0%  -95.53% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                  2353.08Ki ±   0%   42.97Ki ±   0%  -98.17% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                               1033.0Ki ±   0%   481.1Ki ±   0%  -53.42% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                     1020.0Ki ±   0%   481.6Ki ±   0%  -52.78% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                  1576.8Ki ±   0%   809.9Ki ±   0%  -48.64% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                  6.477Mi ±   0%   3.711Mi ±   0%  -42.71% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                              19.289Mi ±   0%   9.859Mi ±   2%  -48.89% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                    19.323Mi ±   1%   9.859Mi ±   2%  -48.98% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                  26.33Mi ±   1%   16.42Mi ±   2%  -37.64% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                 86.67Mi ± 397%   75.70Mi ±   2%  -12.66% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                       24.195Ki ±   0%   8.283Ki ±   0%  -65.77% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step             25.937Ki ±   0%   8.283Ki ±   0%  -68.06% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps          222.61Ki ±   0%   11.68Ki ±   0%  -94.75% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps        2010.15Ki ±   0%   39.92Ki ±   0%  -98.01% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                     1280.2Ki ±   0%   550.2Ki ±   0%  -57.03% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step           1268.3Ki ±   0%   549.8Ki ±   0%  -56.65% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps        1824.5Ki ±   0%   878.4Ki ±   0%  -51.85% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps        6.399Mi ±   0%   3.455Mi ±   0%  -46.01% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                     24.01Mi ±   0%   11.28Mi ±   1%  -53.01% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step           24.24Mi ±   1%   11.23Mi ±   2%  -53.67% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps        31.15Mi ±   0%   17.77Mi ±   2%  -42.97% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps       84.64Mi ± 406%   69.85Mi ±   2%  -17.48% (p=0.002 n=6)
geomean                                                                771.2Ki          318.7Ki         -58.67%

                                                                 │ /tmp/standard.8oda │        /tmp/streaming.45b6         │
                                                                 │     allocs/op      │  allocs/op   vs base               │
Query/a_1,_instant_query                                                  105.00 ± 0%    52.00 ± 0%  -50.48% (p=0.002 n=6)
Query/a_1,_range_query_with_1_step                                        111.00 ± 0%    52.00 ± 0%  -53.15% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                     115.00 ± 0%    56.00 ± 0%  -51.30% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                    143.00 ± 0%    84.00 ± 0%  -41.26% (p=0.002 n=6)
Query/a_100,_instant_query                                                 805.0 ± 0%    745.0 ± 0%   -7.45% (p=0.002 n=6)
Query/a_100,_range_query_with_1_step                                       811.0 ± 0%    745.0 ± 0%   -8.14% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                   1.112k ± 0%   1.046k ± 0%   -5.94% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                  3.615k ± 0%   3.549k ± 0%   -1.83% (p=0.002 n=6)
Query/a_2000,_instant_query                                               14.11k ± 0%   14.05k ± 0%   -0.43% (p=0.002 n=6)
Query/a_2000,_range_query_with_1_step                                     14.12k ± 0%   14.05k ± 0%   -0.46% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                  20.11k ± 0%   20.05k ± 0%   -0.33% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                 70.07k ± 0%   70.01k ± 0%   -0.09% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                        148.00 ± 0%    61.00 ± 0%  -58.78% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1_step                              154.00 ± 0%    61.00 ± 0%  -60.39% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                           158.00 ± 0%    65.00 ± 0%  -58.86% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                          183.00 ± 0%    90.00 ± 0%  -50.82% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                      1251.0 ± 0%    853.0 ± 0%  -31.81% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1_step                            1257.0 ± 0%    853.0 ± 0%  -32.14% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                         1.558k ± 0%   1.154k ± 0%  -25.93% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                        3.761k ± 0%   3.357k ± 0%  -10.74% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                     22.16k ± 0%   16.06k ± 0%  -27.53% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1_step                           22.16k ± 0%   16.06k ± 0%  -27.54% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                        28.16k ± 0%   22.05k ± 0%  -21.68% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                       72.13k ± 0%   66.02k ± 0%   -8.47% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                          472.0 ± 0%    378.0 ± 0%  -19.92% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                       28.30k ± 0%   27.89k ± 0%   -1.44% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                      561.8k ± 0%   555.6k ± 0%   -1.09% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                         384.0 ± 0%    287.0 ± 0%  -25.26% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1_step                               390.0 ± 0%    287.0 ± 0%  -26.41% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                            393.0 ± 0%    290.0 ± 0%  -26.21% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                           415.0 ± 0%    312.0 ± 0%  -24.82% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                      23.27k ± 0%   22.86k ± 0%   -1.76% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1_step                            23.27k ± 0%   22.86k ± 0%   -1.78% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                         23.58k ± 0%   23.16k ± 0%   -1.76% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                        25.71k ± 0%   25.28k ± 0%   -1.65% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                     461.8k ± 0%   455.6k ± 0%   -1.33% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1_step                           461.8k ± 0%   455.7k ± 0%   -1.34% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                        467.8k ± 0%   461.7k ± 1%   -1.31% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                       509.8k ± 0%   503.7k ± 1%   -1.20% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                             143.00 ± 0%    61.00 ± 0%  -57.34% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1_step                                   152.00 ± 0%    61.00 ± 0%  -59.87% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                552.00 ± 0%    65.00 ± 0%  -88.22% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                              4180.00 ± 0%    93.00 ± 0%  -97.78% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                            944.0 ± 0%    754.0 ± 0%  -20.13% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1_step                                  854.0 ± 0%    754.0 ± 0%  -11.71% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                              1.551k ± 0%   1.055k ± 0%  -31.98% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                             7.654k ± 0%   3.558k ± 0%  -53.51% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                          16.15k ± 0%   14.06k ± 0%  -12.94% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1_step                                14.16k ± 0%   14.06k ± 0%   -0.71% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                             20.55k ± 0%   20.05k ± 0%   -2.42% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                            74.12k ± 0%   70.01k ± 0%   -5.54% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                       233.0 ± 0%    133.0 ± 0%  -42.92% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1_step                             233.0 ± 0%    133.0 ± 0%  -42.92% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                          762.0 ± 0%    167.0 ± 0%  -78.08% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                        5540.0 ± 0%    445.0 ± 0%  -91.97% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                    9.174k ± 0%   7.867k ± 0%  -14.25% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1_step                          8.291k ± 0%   7.867k ± 0%   -5.11% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                       33.22k ± 0%   11.17k ± 0%  -66.38% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                     257.34k ± 0%   38.68k ± 0%  -84.97% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                   180.3k ± 0%   156.1k ± 0%  -13.38% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1_step                         162.3k ± 0%   156.1k ± 0%   -3.81% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                      631.7k ± 0%   222.1k ± 0%  -64.84% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                    4848.7k ± 1%   771.7k ± 0%  -84.08% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                      259.0 ± 0%    144.0 ± 0%  -44.40% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1_step                            283.0 ± 0%    144.0 ± 0%  -49.12% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                        3293.0 ± 0%    178.0 ± 0%  -94.59% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                      30631.0 ± 0%    456.0 ± 0%  -98.51% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                   8.979k ± 0%   7.771k ± 0%  -13.45% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1_step                         7.915k ± 0%   7.771k ± 0%   -1.81% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                      14.19k ± 0%   11.07k ± 0%  -21.98% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                     68.76k ± 0%   38.58k ± 0%  -43.89% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                  176.2k ± 0%   154.1k ± 0%  -12.54% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1_step                        154.2k ± 0%   154.1k ± 0%   -0.09% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                     223.2k ± 0%   220.0k ± 0%   -1.40% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                    799.9k ± 5%   769.6k ± 0%   -3.78% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                            331.0 ± 0%    152.0 ± 0%  -54.08% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1_step                  348.0 ± 0%    152.0 ± 0%  -56.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps              2266.0 ± 0%    186.0 ± 0%  -91.79% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps            19670.0 ± 0%    431.0 ± 0%  -97.81% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                        13.425k ± 0%   8.879k ± 0%  -33.86% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1_step              12.363k ± 0%   8.879k ± 0%  -28.18% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps            18.64k ± 0%   12.18k ± 0%  -34.65% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps           69.91k ± 0%   36.39k ± 0%  -47.95% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                        264.3k ± 0%   176.1k ± 0%  -33.38% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1_step              242.4k ± 0%   176.1k ± 0%  -27.35% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps           311.3k ± 0%   242.1k ± 0%  -22.25% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps          822.0k ± 5%   725.7k ± 0%  -11.72% (p=0.002 n=6)
geomean                                                                   7.761k        4.300k       -44.59%

Which issue(s) this PR fixes or relates to

(none)

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[jdbaldry]

I'm guessing we defer docs on experimental features until it is stabilized which seems reasonable to me. In that case, don't consider my review necessary for the docs changes to be merged :)

[pracucci]

Super cool work! I'm hyper pumped about this! The design is very clean and easy to follow. I only found a potential critical issue with hash collisions (let me know if I'm missing something). Regardless I may be right or wrong with hash collision issue, I'm approving anyway because:

1. Changes looks safe to me when using the standard engine (I can't see regressions)
2. The new engine is experimental and a WIP. This PR is huge, so I suggest to merge it unless there are significant concerns, and then re-iterate on this through (hopefully) smaller PRs

P.S. I read the code, but I haven't reviewed the tests.

[jhesketh]

Thanks for fixing up the nits. I agree with Marco's comment about it not having any regression for the existing engine, and that we should merge and iterate from there.

[jhesketh]

(FYI I have skimmed the tests and they look good to me. I have not considered what might be missing from them, but they are a great start)