Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve list query #140

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Improve list query #140

wants to merge 2 commits into from

Conversation

marco6
Copy link
Collaborator

@marco6 marco6 commented Jul 22, 2024

This PR changes slightly the index to make it covering when querying for latest rows.

It also changes the List query (and all those related to the same index) so that it is possible to exploit better the properties of this updated index. While doing that, it removes a redundant query.

@marco6
Copy link
Collaborator Author

marco6 commented Jul 22, 2024

Depends on #136, #132 and #131 .

@marco6
Copy link
Collaborator Author

marco6 commented Jul 24, 2024

This PR also depends on #141

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 24, 2024 •
edited
Loading

Benchmark Result

goos: linux
goarch: amd64
pkg: github.com/canonical/k8s-dqlite/test
cpu: AMD EPYC 7763 64-Core Processor                
                                        │    Before     │                After                │
                                        │    sec/op     │    sec/op     vs base               │
Compaction/sqlite-4                        39.55µ ±  9%   48.72µ ± 27%        ~ (p=0.073 n=7)
Compaction/dqlite-4                        42.71µ ±  5%   42.53µ ± 17%        ~ (p=0.902 n=7)
Create/sqlite-4                            1.502m ± 16%   1.365m ± 55%        ~ (p=0.209 n=7)
Create/dqlite-4                            1.243m ± 20%   1.189m ±  9%        ~ (p=0.535 n=7)
Delete/sqlite-4                            1.372m ± 14%   1.319m ± 21%        ~ (p=0.535 n=7)
Delete/dqlite-4                            1.348m ±  8%   1.593m ± 33%  +18.13% (p=0.007 n=7)
Get/sqlite-4                               270.8µ ±  0%   254.0µ ±  1%   -6.21% (p=0.001 n=7)
Get/dqlite-4                               361.5µ ±  2%   342.8µ ±  1%   -5.17% (p=0.001 n=7)
List/sqlite-tiny/all-4                     4.396µ ±  3%   4.064µ ±  3%   -7.55% (p=0.001 n=7)
List/sqlite-tiny/pagination-4              8.884µ ±  4%   5.376µ ±  5%  -39.49% (p=0.001 n=7)
List/sqlite-fits-in-page/all-4             7.440µ ±  3%   5.307µ ±  3%  -28.67% (p=0.001 n=7)
List/sqlite-fits-in-page/pagination-4     22.990µ ±  4%   7.150µ ±  2%  -68.90% (p=0.001 n=7)
List/sqlite-overflows-page/all-4          17.969µ ±  5%   9.573µ ±  3%  -46.72% (p=0.001 n=7)
List/sqlite-overflows-page/pagination-4    48.38µ ±  1%   13.03µ ±  4%  -73.06% (p=0.001 n=7)
List/dqlite-tiny/all-4                     2.788µ ±  3%   2.405µ ± 22%  -13.74% (p=0.001 n=7)
List/dqlite-tiny/pagination-4              5.903µ ±  1%   3.257µ ±  3%  -44.82% (p=0.001 n=7)
List/dqlite-fits-in-page/all-4             5.998µ ±  4%   4.592µ ± 10%  -23.44% (p=0.001 n=7)
List/dqlite-fits-in-page/pagination-4     16.975µ ±  4%   5.943µ ±  3%  -64.99% (p=0.001 n=7)
List/dqlite-overflows-page/all-4           19.22µ ±  3%   12.20µ ±  8%  -36.55% (p=0.001 n=7)
List/dqlite-overflows-page/pagination-4    43.21µ ±  2%   16.31µ ±  7%  -62.25% (p=0.001 n=7)
Update/sqlite-4                            1.654m ± 15%   1.453m ±  9%  -12.12% (p=0.017 n=7)
Update/dqlite-4                            1.616m ± 28%   1.364m ±  9%        ~ (p=0.318 n=7)
geomean                                    66.43µ         46.72µ        -29.67%

                                        │        Before        │                    After                     │
                                        │ page-cache-misses/op │ page-cache-misses/op  vs base                │
Compaction/sqlite-4                               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Compaction/dqlite-4                               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/sqlite-4                                      0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/dqlite-4                                      0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/all-4                            0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/pagination-4                     0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/all-4                    0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/pagination-4             0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/all-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/pagination-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/all-4                            0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/pagination-4                     0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/all-4                    0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/pagination-4             0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/all-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/pagination-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
geomean                                                      ²                         +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                        │        Before        │                    After                     │
                                        │ page-cache-spills/op │ page-cache-spills/op  vs base                │
Compaction/sqlite-4                               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Compaction/dqlite-4                               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/sqlite-4                                      0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/dqlite-4                                      0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/all-4                            0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/pagination-4                     0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/all-4                    0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/pagination-4             0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/all-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/pagination-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/all-4                            0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/pagination-4                     0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/all-4                    0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/pagination-4             0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/all-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/pagination-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/sqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/dqlite-4                                   0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
geomean                                                      ²                         +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                        │    Before     │                After                 │
                                        │ page-reads/op │ page-reads/op  vs base               │
Compaction/sqlite-4                         12.19 ±  1%      12.19 ± 1%        ~ (p=0.834 n=7)
Compaction/dqlite-4                         12.22 ±  1%      12.18 ± 1%        ~ (p=0.600 n=7)
Create/sqlite-4                             13.60 ±  5%      13.97 ± 8%        ~ (p=0.510 n=7)
Create/dqlite-4                             12.83 ±  2%      12.91 ± 1%        ~ (p=0.364 n=7)
Delete/sqlite-4                             25.14 ±  2%      24.86 ± 3%        ~ (p=0.122 n=7)
Delete/dqlite-4                             23.90 ±  1%      23.75 ± 1%        ~ (p=0.477 n=7)
Get/sqlite-4                                14.96 ±  0%      12.97 ± 0%  -13.30% (p=0.001 n=7)
Get/dqlite-4                                11.96 ±  0%      12.98 ± 0%   +8.53% (p=0.001 n=7)
List/sqlite-tiny/all-4                     192.4m ±  0%     191.8m ± 0%   -0.31% (p=0.001 n=7)
List/sqlite-tiny/pagination-4             1132.0m ±  3%     194.9m ± 1%  -82.78% (p=0.001 n=7)
List/sqlite-fits-in-page/all-4             164.8m ±  0%     163.2m ± 0%   -0.97% (p=0.001 n=7)
List/sqlite-fits-in-page/pagination-4      965.7m ±  0%     165.6m ± 0%  -82.85% (p=0.001 n=7)
List/sqlite-overflows-page/all-4           330.6m ± 21%     182.4m ± 0%  -44.83% (p=0.001 n=7)
List/sqlite-overflows-page/pagination-4   2080.0m ±  0%     185.3m ± 0%  -91.09% (p=0.001 n=7)
List/dqlite-tiny/all-4                     191.2m ±  0%     190.7m ± 1%        ~ (p=0.682 n=7)
List/dqlite-tiny/pagination-4             1125.0m ±  3%     191.7m ± 0%  -82.96% (p=0.001 n=7)
List/dqlite-fits-in-page/all-4             163.1m ± 59%     163.0m ± 1%        ~ (p=0.141 n=7)
List/dqlite-fits-in-page/pagination-4      954.3m ±  3%     163.6m ± 0%  -82.86% (p=0.001 n=7)
List/dqlite-overflows-page/all-4           182.8m ±  0%     182.5m ± 0%   -0.16% (p=0.018 n=7)
List/dqlite-overflows-page/pagination-4   1071.0m ±  0%     185.4m ± 0%  -82.69% (p=0.001 n=7)
Update/sqlite-4                             24.22 ±  3%      25.15 ± 2%   +3.84% (p=0.017 n=7)
Update/dqlite-4                             23.72 ±  3%      24.25 ± 2%   +2.23% (p=0.038 n=7)
geomean                                     2.409            1.407       -41.59%

                                        │     Before     │                 After                  │
                                        │ page-writes/op │ page-writes/op  vs base                │
Compaction/sqlite-4                        207.6m ± 0%        208.2m ± 1%       ~ (p=0.087 n=7)
Compaction/dqlite-4                        187.7m ± 1%        188.3m ± 1%       ~ (p=0.057 n=7)
Create/sqlite-4                             5.107 ± 0%         5.116 ± 1%       ~ (p=0.469 n=7)
Create/dqlite-4                             4.136 ± 4%         4.137 ± 2%       ~ (p=0.931 n=7)
Delete/sqlite-4                             5.334 ± 1%         5.307 ± 1%       ~ (p=0.165 n=7)
Delete/dqlite-4                             4.389 ± 1%         4.393 ± 1%       ~ (p=0.456 n=7)
Get/sqlite-4                                0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/dqlite-4                                0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/all-4                      0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-tiny/pagination-4               0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/all-4              0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/pagination-4       0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/all-4            0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/pagination-4     0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/all-4                      0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-tiny/pagination-4               0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/all-4              0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/pagination-4       0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/all-4            0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/pagination-4     0.000 ± 0%         0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/sqlite-4                             5.126 ± 0%         5.143 ± 0%       ~ (p=0.206 n=7)
Update/dqlite-4                             4.151 ± 1%         4.163 ± 1%  +0.29% (p=0.002 n=7)
geomean                                                ²                   +0.05%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                        │     Before     │                 After                 │
                                        │ sec-reading/op │ sec-reading/op  vs base               │
Compaction/sqlite-4                        122.2µ ±   7%    113.6µ ±  24%        ~ (p=0.383 n=7)
Compaction/dqlite-4                        24.10µ ±  42%    19.70µ ±  43%        ~ (p=0.073 n=7)
Create/sqlite-4                           120.30µ ± 350%    65.20µ ± 701%        ~ (p=1.000 n=7)
Create/dqlite-4                            7.600µ ±  63%   10.300µ ±  52%        ~ (p=0.456 n=7)
Delete/sqlite-4                            172.3µ ± 164%    330.9µ ±  74%        ~ (p=0.259 n=7)
Delete/dqlite-4                            55.10µ ±  31%    33.60µ ±  27%  -39.02% (p=0.001 n=7)
Get/sqlite-4                               87.40µ ±   6%    78.50µ ±   9%  -10.18% (p=0.001 n=7)
Get/dqlite-4                               32.30µ ±  10%    25.00µ ±  11%  -22.60% (p=0.001 n=7)
List/sqlite-tiny/all-4                     14.10µ ±   2%    10.50µ ±   5%  -25.53% (p=0.001 n=7)
List/sqlite-tiny/pagination-4              19.40µ ±   7%    12.00µ ±   3%  -38.14% (p=0.001 n=7)
List/sqlite-fits-in-page/all-4             19.80µ ±  25%    20.20µ ±   8%        ~ (p=0.686 n=7)
List/sqlite-fits-in-page/pagination-4      63.40µ ±   7%    25.50µ ±  19%  -59.78% (p=0.001 n=7)
List/sqlite-overflows-page/all-4           70.90µ ±  14%    54.70µ ±   8%  -22.85% (p=0.001 n=7)
List/sqlite-overflows-page/pagination-4   180.20µ ±   3%    57.80µ ±  15%  -67.92% (p=0.001 n=7)
List/dqlite-tiny/all-4                     4.700µ ±  23%    3.500µ ±  49%        ~ (p=0.052 n=7)
List/dqlite-tiny/pagination-4              8.300µ ±  19%    5.100µ ±   4%  -38.55% (p=0.001 n=7)
List/dqlite-fits-in-page/all-4            10.200µ ±  20%    7.500µ ±  25%  -26.47% (p=0.001 n=7)
List/dqlite-fits-in-page/pagination-4     24.800µ ±   9%    8.200µ ±  18%  -66.94% (p=0.001 n=7)
List/dqlite-overflows-page/all-4           33.60µ ±   5%    19.80µ ±  14%  -41.07% (p=0.001 n=7)
List/dqlite-overflows-page/pagination-4    75.20µ ±  13%    24.70µ ±  11%  -67.15% (p=0.001 n=7)
Update/sqlite-4                            187.4µ ±  57%    184.0µ ± 130%        ~ (p=0.456 n=7)
Update/dqlite-4                            93.20µ ±  21%   100.70µ ±  19%        ~ (p=0.259 n=7)
geomean                                    39.56µ           27.97µ         -29.31%

                                        │     Before      │                  After                  │
                                        │ sec-writing/op  │ sec-writing/op  vs base                 │
Compaction/sqlite-4                        10.40µ ±  7%       10.50µ ±  9%        ~ (p=0.303 n=7)
Compaction/dqlite-4                        6.600µ ±  8%       6.600µ ±  6%        ~ (p=0.721 n=7)
Create/sqlite-4                            1.259m ± 20%       1.107m ± 70%        ~ (p=0.259 n=7)
Create/dqlite-4                            29.20µ ± 33%       36.60µ ± 40%        ~ (p=0.128 n=7)
Delete/sqlite-4                           1001.0µ ± 17%       954.9µ ± 28%        ~ (p=0.535 n=7)
Delete/dqlite-4                            23.40µ ± 33%       27.90µ ± 37%        ~ (p=0.165 n=7)
Get/sqlite-4                                0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Get/dqlite-4                                0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-tiny/all-4                      0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-tiny/pagination-4               0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/all-4              0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-fits-in-page/pagination-4       0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/all-4            0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/sqlite-overflows-page/pagination-4     0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-tiny/all-4                      0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-tiny/pagination-4               0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/all-4              0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/pagination-4       0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/all-4            0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
List/dqlite-overflows-page/pagination-4     0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Update/sqlite-4                           1228.0µ ± 22%       989.1µ ± 14%  -19.45% (p=0.017 n=7)
Update/dqlite-4                            17.80µ ± 58%       29.90µ ± 22%  +67.98% (p=0.004 n=7)
geomean                                                 ²                    +2.47%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                        │     Before     │                 After                 │
                                        │      B/op      │      B/op       vs base               │
Compaction/sqlite-4                         1005.0 ± 21%     524.0 ± 115%        ~ (p=0.259 n=7)
Compaction/dqlite-4                        3.854Ki ± 54%   2.931Ki ±  61%        ~ (p=0.128 n=7)
Create/sqlite-4                            21.17Ki ±  4%   21.32Ki ±   5%        ~ (p=0.966 n=7)
Create/dqlite-4                            20.12Ki ±  0%   20.11Ki ±   0%        ~ (p=0.535 n=7)
Delete/sqlite-4                            32.65Ki ±  2%   32.25Ki ±   3%        ~ (p=0.128 n=7)
Delete/dqlite-4                            29.47Ki ±  0%   29.50Ki ±   0%   +0.09% (p=0.041 n=7)
Get/sqlite-4                               25.08Ki ±  0%   24.41Ki ±   0%   -2.67% (p=0.001 n=7)
Get/dqlite-4                               22.89Ki ±  0%   22.26Ki ±   0%   -2.77% (p=0.001 n=7)
List/sqlite-tiny/all-4                     1.266Ki ±  3%   1.203Ki ±   2%   -4.94% (p=0.001 n=7)
List/sqlite-tiny/pagination-4              1.837Ki ±  5%   1.355Ki ±   3%  -26.21% (p=0.001 n=7)
List/sqlite-fits-in-page/all-4             5.099Ki ±  3%   4.572Ki ±   2%  -10.32% (p=0.001 n=7)
List/sqlite-fits-in-page/pagination-4     10.222Ki ±  0%   5.137Ki ±   3%  -49.75% (p=0.001 n=7)
List/sqlite-overflows-page/all-4           27.45Ki ±  5%   20.70Ki ±   3%  -24.59% (p=0.001 n=7)
List/sqlite-overflows-page/pagination-4    46.16Ki ±  0%   22.90Ki ±   2%  -50.38% (p=0.001 n=7)
List/dqlite-tiny/all-4                       873.0 ±  3%     844.0 ±  18%        ~ (p=0.684 n=7)
List/dqlite-tiny/pagination-4                847.0 ±  1%     854.0 ±   1%   +0.83% (p=0.036 n=7)
List/dqlite-fits-in-page/all-4             4.264Ki ±  2%   4.249Ki ±  10%        ~ (p=0.334 n=7)
List/dqlite-fits-in-page/pagination-4      4.365Ki ±  2%   4.291Ki ±   5%        ~ (p=0.259 n=7)
List/dqlite-overflows-page/all-4           21.17Ki ±  1%   20.78Ki ±   5%   -1.85% (p=0.002 n=7)
List/dqlite-overflows-page/pagination-4    22.78Ki ±  8%   22.87Ki ±   4%        ~ (p=0.197 n=7)
Update/sqlite-4                            30.26Ki ±  1%   30.32Ki ±   1%        ~ (p=0.710 n=7)
Update/dqlite-4                            27.65Ki ±  0%   27.62Ki ±   0%   -0.09% (p=0.011 n=7)
geomean                                    8.875Ki         7.681Ki         -13.46%

                                        │   Before    │                 After                 │
                                        │  allocs/op  │  allocs/op    vs base                 │
Compaction/sqlite-4                       39.00 ± 21%   18.00 ± 139%        ~ (p=0.220 n=7)
Compaction/dqlite-4                       93.00 ± 55%   70.00 ±  63%        ~ (p=0.128 n=7)
Create/sqlite-4                           449.0 ±  6%   452.0 ±   7%        ~ (p=0.731 n=7)
Create/dqlite-4                           372.0 ±  0%   372.0 ±   0%        ~ (p=1.000 n=7)
Delete/sqlite-4                           727.0 ±  3%   716.0 ±   3%        ~ (p=0.177 n=7)
Delete/dqlite-4                           510.0 ±  0%   510.0 ±   0%        ~ (p=0.392 n=7)
Get/sqlite-4                              563.0 ±  0%   563.0 ±   0%        ~ (p=1.000 n=7) ¹
Get/dqlite-4                              407.0 ±  0%   407.0 ±   0%        ~ (p=1.000 n=7) ¹
List/sqlite-tiny/all-4                    41.00 ±  5%   39.00 ±   3%   -4.88% (p=0.001 n=7)
List/sqlite-tiny/pagination-4             63.00 ±  3%   45.00 ±   4%  -28.57% (p=0.001 n=7)
List/sqlite-fits-in-page/all-4            47.00 ±  4%   40.00 ±   3%  -14.89% (p=0.001 n=7)
List/sqlite-fits-in-page/pagination-4     93.00 ±  0%   47.00 ±   4%  -49.46% (p=0.001 n=7)
List/sqlite-overflows-page/all-4          58.00 ±  5%   41.00 ±   5%  -29.31% (p=0.001 n=7)
List/sqlite-overflows-page/pagination-4   93.00 ±  0%   48.00 ±   2%  -48.39% (p=0.001 n=7)
List/dqlite-tiny/all-4                    14.00 ±  7%   13.00 ±  23%        ~ (p=0.462 n=7)
List/dqlite-tiny/pagination-4             13.00 ±  0%   13.00 ±   0%        ~ (p=1.000 n=7) ¹
List/dqlite-fits-in-page/all-4            18.00 ±  6%   17.00 ±  12%        ~ (p=0.331 n=7)
List/dqlite-fits-in-page/pagination-4     18.00 ±  6%   17.00 ±  12%        ~ (p=0.592 n=7)
List/dqlite-overflows-page/all-4          29.00 ±  3%   28.00 ±   7%   -3.45% (p=0.033 n=7)
List/dqlite-overflows-page/pagination-4   32.00 ± 12%   33.00 ±   6%   +3.12% (p=0.022 n=7)
Update/sqlite-4                           667.0 ±  1%   669.0 ±   1%        ~ (p=0.738 n=7)
Update/dqlite-4                           491.0 ±  0%   492.0 ±   0%        ~ (p=0.070 n=7)
geomean                                   97.41         83.11         -14.68%
¹ all samples are equal

                                        │        Before         │                    After                     │
                                        │ network-bytes-read/op │ network-bytes-read/op  vs base               │
Compaction/dqlite-4                               1.393Ki ± 56%           1.049Ki ± 63%        ~ (p=0.128 n=7)
Create/dqlite-4                                     239.7 ±  1%             239.8 ±  0%        ~ (p=0.591 n=7)
Delete/dqlite-4                                   1.158Ki ±  0%           1.157Ki ±  1%   -0.08% (p=0.012 n=7)
Get/dqlite-4                                       1160.0 ±  0%             984.4 ±  0%  -15.14% (p=0.001 n=7)
List/dqlite-tiny/all-4                              203.5 ±  5%             193.4 ± 29%        ~ (p=0.710 n=7)
List/dqlite-tiny/pagination-4                       189.4 ±  1%             195.7 ±  1%   +3.33% (p=0.001 n=7)
List/dqlite-fits-in-page/all-4                    1.215Ki ±  4%           1.210Ki ± 16%        ~ (p=0.318 n=7)
List/dqlite-fits-in-page/pagination-4             1.234Ki ±  3%           1.211Ki ±  9%        ~ (p=0.596 n=7)
List/dqlite-overflows-page/all-4                  6.210Ki ±  2%           6.030Ki ±  8%   -2.89% (p=0.002 n=7)
List/dqlite-overflows-page/pagination-4           6.874Ki ± 14%           6.958Ki ±  5%        ~ (p=0.067 n=7)
Update/dqlite-4                                     564.0 ±  0%             564.5 ±  0%   +0.09% (p=0.035 n=7)
geomean                                             970.5                   926.9         -4.49%

                                        │          Before          │                      After                      │
                                        │ network-bytes-written/op │ network-bytes-written/op  vs base               │
Compaction/dqlite-4                                    4.352 ±  7%                4.263 ± 14%        ~ (p=0.620 n=7)
Create/dqlite-4                                        112.6 ±  3%                112.8 ±  1%        ~ (p=0.477 n=7)
Delete/dqlite-4                                        423.7 ±  0%                423.4 ±  0%   -0.07% (p=0.048 n=7)
Get/dqlite-4                                           80.50 ±  0%                80.48 ±  0%   -0.02% (p=0.031 n=7)
List/dqlite-tiny/all-4                               0.04314 ±  9%              0.03900 ± 59%        ~ (p=0.710 n=7)
List/dqlite-tiny/pagination-4                        0.04633 ±  8%              0.04362 ±  3%        ~ (p=0.097 n=7)
List/dqlite-fits-in-page/all-4                       0.05136 ±  6%              0.04994 ± 27%        ~ (p=0.073 n=7)
List/dqlite-fits-in-page/pagination-4                0.08210 ± 17%              0.05606 ± 12%  -31.72% (p=0.001 n=7)
List/dqlite-overflows-page/all-4                     0.06480 ±  8%              0.05973 ± 13%   -7.82% (p=0.001 n=7)
List/dqlite-overflows-page/pagination-4              0.10950 ± 27%              0.08913 ± 11%  -18.60% (p=0.001 n=7)
Update/dqlite-4                                        263.3 ±  0%                263.3 ±  0%        ~ (p=0.386 n=7)
geomean                                                1.660                      1.532         -7.67%
Current status 
goos: linux
goarch: amd64
pkg: github.com/canonical/k8s-dqlite/test
cpu: AMD EPYC 7763 64-Core Processor                
                                        │    After     │
                                        │    sec/op    │
Compaction/sqlite-4                       48.72µ ± 27%
Compaction/dqlite-4                       42.53µ ± 17%
Create/sqlite-4                           1.365m ± 55%
Create/dqlite-4                           1.189m ±  9%
Delete/sqlite-4                           1.319m ± 21%
Delete/dqlite-4                           1.593m ± 33%
Get/sqlite-4                              254.0µ ±  1%
Get/dqlite-4                              342.8µ ±  1%
List/sqlite-tiny/all-4                    4.064µ ±  3%
List/sqlite-tiny/pagination-4             5.376µ ±  5%
List/sqlite-fits-in-page/all-4            5.307µ ±  3%
List/sqlite-fits-in-page/pagination-4     7.150µ ±  2%
List/sqlite-overflows-page/all-4          9.573µ ±  3%
List/sqlite-overflows-page/pagination-4   13.03µ ±  4%
List/dqlite-tiny/all-4                    2.405µ ± 22%
List/dqlite-tiny/pagination-4             3.257µ ±  3%
List/dqlite-fits-in-page/all-4            4.592µ ± 10%
List/dqlite-fits-in-page/pagination-4     5.943µ ±  3%
List/dqlite-overflows-page/all-4          12.20µ ±  8%
List/dqlite-overflows-page/pagination-4   16.31µ ±  7%
Update/sqlite-4                           1.453m ±  9%
Update/dqlite-4                           1.364m ±  9%
geomean                                   46.72µ

                                        │        After         │
                                        │ page-cache-misses/op │
Compaction/sqlite-4                               0.000 ± 0%
Compaction/dqlite-4                               0.000 ± 0%
Create/sqlite-4                                   0.000 ± 0%
Create/dqlite-4                                   0.000 ± 0%
Delete/sqlite-4                                   0.000 ± 0%
Delete/dqlite-4                                   0.000 ± 0%
Get/sqlite-4                                      0.000 ± 0%
Get/dqlite-4                                      0.000 ± 0%
List/sqlite-tiny/all-4                            0.000 ± 0%
List/sqlite-tiny/pagination-4                     0.000 ± 0%
List/sqlite-fits-in-page/all-4                    0.000 ± 0%
List/sqlite-fits-in-page/pagination-4             0.000 ± 0%
List/sqlite-overflows-page/all-4                  0.000 ± 0%
List/sqlite-overflows-page/pagination-4           0.000 ± 0%
List/dqlite-tiny/all-4                            0.000 ± 0%
List/dqlite-tiny/pagination-4                     0.000 ± 0%
List/dqlite-fits-in-page/all-4                    0.000 ± 0%
List/dqlite-fits-in-page/pagination-4             0.000 ± 0%
List/dqlite-overflows-page/all-4                  0.000 ± 0%
List/dqlite-overflows-page/pagination-4           0.000 ± 0%
Update/sqlite-4                                   0.000 ± 0%
Update/dqlite-4                                   0.000 ± 0%
geomean                                                      ¹
¹ summaries must be >0 to compute geomean

                                        │        After         │
                                        │ page-cache-spills/op │
Compaction/sqlite-4                               0.000 ± 0%
Compaction/dqlite-4                               0.000 ± 0%
Create/sqlite-4                                   0.000 ± 0%
Create/dqlite-4                                   0.000 ± 0%
Delete/sqlite-4                                   0.000 ± 0%
Delete/dqlite-4                                   0.000 ± 0%
Get/sqlite-4                                      0.000 ± 0%
Get/dqlite-4                                      0.000 ± 0%
List/sqlite-tiny/all-4                            0.000 ± 0%
List/sqlite-tiny/pagination-4                     0.000 ± 0%
List/sqlite-fits-in-page/all-4                    0.000 ± 0%
List/sqlite-fits-in-page/pagination-4             0.000 ± 0%
List/sqlite-overflows-page/all-4                  0.000 ± 0%
List/sqlite-overflows-page/pagination-4           0.000 ± 0%
List/dqlite-tiny/all-4                            0.000 ± 0%
List/dqlite-tiny/pagination-4                     0.000 ± 0%
List/dqlite-fits-in-page/all-4                    0.000 ± 0%
List/dqlite-fits-in-page/pagination-4             0.000 ± 0%
List/dqlite-overflows-page/all-4                  0.000 ± 0%
List/dqlite-overflows-page/pagination-4           0.000 ± 0%
Update/sqlite-4                                   0.000 ± 0%
Update/dqlite-4                                   0.000 ± 0%
geomean                                                      ¹
¹ summaries must be >0 to compute geomean

                                        │     After     │
                                        │ page-reads/op │
Compaction/sqlite-4                          12.19 ± 1%
Compaction/dqlite-4                          12.18 ± 1%
Create/sqlite-4                              13.97 ± 8%
Create/dqlite-4                              12.91 ± 1%
Delete/sqlite-4                              24.86 ± 3%
Delete/dqlite-4                              23.75 ± 1%
Get/sqlite-4                                 12.97 ± 0%
Get/dqlite-4                                 12.98 ± 0%
List/sqlite-tiny/all-4                      191.8m ± 0%
List/sqlite-tiny/pagination-4               194.9m ± 1%
List/sqlite-fits-in-page/all-4              163.2m ± 0%
List/sqlite-fits-in-page/pagination-4       165.6m ± 0%
List/sqlite-overflows-page/all-4            182.4m ± 0%
List/sqlite-overflows-page/pagination-4     185.3m ± 0%
List/dqlite-tiny/all-4                      190.7m ± 1%
List/dqlite-tiny/pagination-4               191.7m ± 0%
List/dqlite-fits-in-page/all-4              163.0m ± 1%
List/dqlite-fits-in-page/pagination-4       163.6m ± 0%
List/dqlite-overflows-page/all-4            182.5m ± 0%
List/dqlite-overflows-page/pagination-4     185.4m ± 0%
Update/sqlite-4                              25.15 ± 2%
Update/dqlite-4                              24.25 ± 2%
geomean                                      1.407

                                        │     After      │
                                        │ page-writes/op │
Compaction/sqlite-4                        208.2m ± 1%
Compaction/dqlite-4                        188.3m ± 1%
Create/sqlite-4                             5.116 ± 1%
Create/dqlite-4                             4.137 ± 2%
Delete/sqlite-4                             5.307 ± 1%
Delete/dqlite-4                             4.393 ± 1%
Get/sqlite-4                                0.000 ± 0%
Get/dqlite-4                                0.000 ± 0%
List/sqlite-tiny/all-4                      0.000 ± 0%
List/sqlite-tiny/pagination-4               0.000 ± 0%
List/sqlite-fits-in-page/all-4              0.000 ± 0%
List/sqlite-fits-in-page/pagination-4       0.000 ± 0%
List/sqlite-overflows-page/all-4            0.000 ± 0%
List/sqlite-overflows-page/pagination-4     0.000 ± 0%
List/dqlite-tiny/all-4                      0.000 ± 0%
List/dqlite-tiny/pagination-4               0.000 ± 0%
List/dqlite-fits-in-page/all-4              0.000 ± 0%
List/dqlite-fits-in-page/pagination-4       0.000 ± 0%
List/dqlite-overflows-page/all-4            0.000 ± 0%
List/dqlite-overflows-page/pagination-4     0.000 ± 0%
Update/sqlite-4                             5.143 ± 0%
Update/dqlite-4                             4.163 ± 1%
geomean                                                ¹
¹ summaries must be >0 to compute geomean

                                        │     After      │
                                        │ sec-reading/op │
Compaction/sqlite-4                        113.6µ ±  24%
Compaction/dqlite-4                        19.70µ ±  43%
Create/sqlite-4                            65.20µ ± 701%
Create/dqlite-4                            10.30µ ±  52%
Delete/sqlite-4                            330.9µ ±  74%
Delete/dqlite-4                            33.60µ ±  27%
Get/sqlite-4                               78.50µ ±   9%
Get/dqlite-4                               25.00µ ±  11%
List/sqlite-tiny/all-4                     10.50µ ±   5%
List/sqlite-tiny/pagination-4              12.00µ ±   3%
List/sqlite-fits-in-page/all-4             20.20µ ±   8%
List/sqlite-fits-in-page/pagination-4      25.50µ ±  19%
List/sqlite-overflows-page/all-4           54.70µ ±   8%
List/sqlite-overflows-page/pagination-4    57.80µ ±  15%
List/dqlite-tiny/all-4                     3.500µ ±  49%
List/dqlite-tiny/pagination-4              5.100µ ±   4%
List/dqlite-fits-in-page/all-4             7.500µ ±  25%
List/dqlite-fits-in-page/pagination-4      8.200µ ±  18%
List/dqlite-overflows-page/all-4           19.80µ ±  14%
List/dqlite-overflows-page/pagination-4    24.70µ ±  11%
Update/sqlite-4                            184.0µ ± 130%
Update/dqlite-4                            100.7µ ±  19%
geomean                                    27.97µ

                                        │     After      │
                                        │ sec-writing/op │
Compaction/sqlite-4                       10.50µ ±  9%
Compaction/dqlite-4                       6.600µ ±  6%
Create/sqlite-4                           1.107m ± 70%
Create/dqlite-4                           36.60µ ± 40%
Delete/sqlite-4                           954.9µ ± 28%
Delete/dqlite-4                           27.90µ ± 37%
Get/sqlite-4                               0.000 ±  0%
Get/dqlite-4                               0.000 ±  0%
List/sqlite-tiny/all-4                     0.000 ±  0%
List/sqlite-tiny/pagination-4              0.000 ±  0%
List/sqlite-fits-in-page/all-4             0.000 ±  0%
List/sqlite-fits-in-page/pagination-4      0.000 ±  0%
List/sqlite-overflows-page/all-4           0.000 ±  0%
List/sqlite-overflows-page/pagination-4    0.000 ±  0%
List/dqlite-tiny/all-4                     0.000 ±  0%
List/dqlite-tiny/pagination-4              0.000 ±  0%
List/dqlite-fits-in-page/all-4             0.000 ±  0%
List/dqlite-fits-in-page/pagination-4      0.000 ±  0%
List/dqlite-overflows-page/all-4           0.000 ±  0%
List/dqlite-overflows-page/pagination-4    0.000 ±  0%
Update/sqlite-4                           989.1µ ± 14%
Update/dqlite-4                           29.90µ ± 22%
geomean                                                ¹
¹ summaries must be >0 to compute geomean

                                        │     After      │
                                        │      B/op      │
Compaction/sqlite-4                         524.0 ± 115%
Compaction/dqlite-4                       2.931Ki ±  61%
Create/sqlite-4                           21.32Ki ±   5%
Create/dqlite-4                           20.11Ki ±   0%
Delete/sqlite-4                           32.25Ki ±   3%
Delete/dqlite-4                           29.50Ki ±   0%
Get/sqlite-4                              24.41Ki ±   0%
Get/dqlite-4                              22.26Ki ±   0%
List/sqlite-tiny/all-4                    1.203Ki ±   2%
List/sqlite-tiny/pagination-4             1.355Ki ±   3%
List/sqlite-fits-in-page/all-4            4.572Ki ±   2%
List/sqlite-fits-in-page/pagination-4     5.137Ki ±   3%
List/sqlite-overflows-page/all-4          20.70Ki ±   3%
List/sqlite-overflows-page/pagination-4   22.90Ki ±   2%
List/dqlite-tiny/all-4                      844.0 ±  18%
List/dqlite-tiny/pagination-4               854.0 ±   1%
List/dqlite-fits-in-page/all-4            4.249Ki ±  10%
List/dqlite-fits-in-page/pagination-4     4.291Ki ±   5%
List/dqlite-overflows-page/all-4          20.78Ki ±   5%
List/dqlite-overflows-page/pagination-4   22.87Ki ±   4%
Update/sqlite-4                           30.32Ki ±   1%
Update/dqlite-4                           27.62Ki ±   0%
geomean                                   7.681Ki

                                        │    After     │
                                        │  allocs/op   │
Compaction/sqlite-4                       18.00 ± 139%
Compaction/dqlite-4                       70.00 ±  63%
Create/sqlite-4                           452.0 ±   7%
Create/dqlite-4                           372.0 ±   0%
Delete/sqlite-4                           716.0 ±   3%
Delete/dqlite-4                           510.0 ±   0%
Get/sqlite-4                              563.0 ±   0%
Get/dqlite-4                              407.0 ±   0%
List/sqlite-tiny/all-4                    39.00 ±   3%
List/sqlite-tiny/pagination-4             45.00 ±   4%
List/sqlite-fits-in-page/all-4            40.00 ±   3%
List/sqlite-fits-in-page/pagination-4     47.00 ±   4%
List/sqlite-overflows-page/all-4          41.00 ±   5%
List/sqlite-overflows-page/pagination-4   48.00 ±   2%
List/dqlite-tiny/all-4                    13.00 ±  23%
List/dqlite-tiny/pagination-4             13.00 ±   0%
List/dqlite-fits-in-page/all-4            17.00 ±  12%
List/dqlite-fits-in-page/pagination-4     17.00 ±  12%
List/dqlite-overflows-page/all-4          28.00 ±   7%
List/dqlite-overflows-page/pagination-4   33.00 ±   6%
Update/sqlite-4                           669.0 ±   1%
Update/dqlite-4                           492.0 ±   0%
geomean                                   83.11

                                        │         After         │
                                        │ network-bytes-read/op │
Compaction/dqlite-4                               1.049Ki ± 63%
Create/dqlite-4                                     239.8 ±  0%
Delete/dqlite-4                                   1.157Ki ±  1%
Get/dqlite-4                                        984.4 ±  0%
List/dqlite-tiny/all-4                              193.4 ± 29%
List/dqlite-tiny/pagination-4                       195.7 ±  1%
List/dqlite-fits-in-page/all-4                    1.210Ki ± 16%
List/dqlite-fits-in-page/pagination-4             1.211Ki ±  9%
List/dqlite-overflows-page/all-4                  6.030Ki ±  8%
List/dqlite-overflows-page/pagination-4           6.958Ki ±  5%
Update/dqlite-4                                     564.5 ±  0%
geomean                                             926.9

                                        │          After           │
                                        │ network-bytes-written/op │
Compaction/dqlite-4                                    4.263 ± 14%
Create/dqlite-4                                        112.8 ±  1%
Delete/dqlite-4                                        423.4 ±  0%
Get/dqlite-4                                           80.48 ±  0%
List/dqlite-tiny/all-4                               0.03900 ± 59%
List/dqlite-tiny/pagination-4                        0.04362 ±  3%
List/dqlite-fits-in-page/all-4                       0.04994 ± 27%
List/dqlite-fits-in-page/pagination-4                0.05606 ± 12%
List/dqlite-overflows-page/all-4                     0.05973 ± 13%
List/dqlite-overflows-page/pagination-4              0.08913 ± 11%
Update/dqlite-4                                        263.3 ±  0%
geomean                                                1.532

@marco6 marco6 mentioned this pull request Jul 29, 2024
Merged
@marco6 marco6 force-pushed the marco6/improve-list-sql branch 2 times, most recently from 8163d25 to a683b84 Compare August 1, 2024 12:50
@marco6
Copy link
Collaborator Author

marco6 commented Sep 6, 2024

During the development of #170 we found out that List queries cannot return PrevKV. Moreover, Update does not use get to retrieve the event anymore.

Delete query is on its way to be updated as well. This means that all GET path do not really need the prevKV and we could improve the read path way more by not reading the previous value (less data to read, less allocations).

There is still a place where this might happen: the watch query. Watchers might ask for prevKV. In this case we need to understand how to deal with that request (if necessary).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@louiseschmidtgen louiseschmidtgen louiseschmidtgen left review comments

Assignees
No one assigned
Labels
next-release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marco6 @louiseschmidtgen