Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: Replace TwigNode vector with BTreeMap #82

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

arriqaaq
Copy link
Contributor

@arriqaaq arriqaaq commented Jan 31, 2025

Description

The current issue with versioning was due to the usage of vector inside a twignode to maintain all versions. As the vector grows larger, this was taking significant time and became worse as reported during surrealkv reload. This PR replaces the vector with a btreemap, and adds a reverse map from timestamp to version to make timestamp queries faster. There is a significant improvement as you can see.

But this comes at a cost of slow down in insertion time. Keeping the PR for future reference.

on main:

Insertion time for 1M key-version pairs: 4.630727792s

Each query type will be executed 1 times
Tree contains 100 keys with 10000 versions each

Results:
Query Type                Total Time      Avg Time  
--------------------------------------------------
LatestByVersion           64.542µs 64.542µs
LatestByTs                7.375µs 7.375µs
LastLessThanTs            3.917µs 3.917µs
LastLessOrEqualTs         3.042µs 3.042µs
FirstGreaterThanTs        7.125µs 7.125µs
FirstGreaterOrEqualTs     3.166µs 3.166µs

on this branch:

Insertion time for 1M key-version pairs: 81.642150042s

Each query type will be executed 1 times
Tree contains 100 keys with 10000 versions each

Results:
Query Type                Total Time      Avg Time  
--------------------------------------------------
LatestByVersion           229.25µs 229.25µs
LatestByTs                186.625µs 186.625µs
LastLessThanTs            182.292µs 182.292µs
LastLessOrEqualTs         180.875µs 180.875µs
FirstGreaterThanTs        173.625µs 173.625µs
FirstGreaterOrEqualTs     172.917µs 172.917µs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant