Skip to content

Commit

Permalink
Refactor and Add SkipSet
Browse files Browse the repository at this point in the history
Refactor implementation and add more convenient APIs
  • Loading branch information
al8n committed May 3, 2024
1 parent fa55ff6 commit 37ceb55
Show file tree
Hide file tree
Showing 39 changed files with 7,005 additions and 2,790 deletions.
24 changes: 24 additions & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
codecov:
require_ci_to_pass: false

ignore:
- integration/

coverage:
status:
project: # Overall project status
default:
target: auto
if_not_found: success
only_pulls: false
patch: # Status for the patch in pull requests
default:
target: auto
if_not_found: success
only_pulls: true
changes: false # Whether to comment on the coverage changes in pull requests

comment:
layout: "header, diff, files, footer"
behavior: default
require_changes: false
40 changes: 19 additions & 21 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ on:
- "LICENSE-*"
- "**.md"
- "**.txt"
workflow_dispatch:

env:
CARGO_TERM_COLOR: always
Expand Down Expand Up @@ -130,7 +131,7 @@ jobs:
- name: cargo build --target ${{ matrix.target }}
run: |
rustup target add ${{ matrix.target }}
cargo build --features js --target ${{ matrix.target }}
cargo build --target ${{ matrix.target }}
if: matrix.target == 'wasm32-unknown-unknown'
# - name: cargo build --target ${{ matrix.target }}
# run: |
Expand All @@ -149,7 +150,7 @@ jobs:
- name: Install rust-src
run: rustup component add rust-src
- name: ASAN / LSAN / TSAN
run: . ci/sanitizer.sh
run: ci/sanitizer.sh

# valgrind
valgrind:
Expand All @@ -169,21 +170,20 @@ jobs:
sudo apt-get update -y
sudo apt-get install -y valgrind
# Compile tests
- name: cargo build test-fixed
run: cargo build --bin test-fixed
working-directory: integration

# Run with valgrind
- name: Run valgrind test-vec
run: valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./target/debug/test-vec
- name: build & valgrind (vec backend)
run: |
cargo build --bin test-vec --target-dir .
valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./debug/test-vec
working-directory: integration

- name: Run valgrind test-mmap
run: valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./target/debug/test-mmap
- name: build & valgrind (mmap backend)
run: |
cargo build --bin test-mmap --target-dir .
valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./debug/test-mmap
working-directory: integration

- name: Run valgrind test-mmap-anon
run: valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./target/debug/test-mmap-anon
- name: build & valgrind (mmap-anon backend)
run: |
cargo build --bin test-mmap-anon --target-dir .
valgrind --error-exitcode=1 --leak-check=full --show-leak-kinds=all ./debug/test-mmap-anon
working-directory: integration


Expand All @@ -195,16 +195,15 @@ jobs:
- name: Miri
run: ci/miri.sh

# Loom
# loom:
# name: loom
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v3
# - uses: actions/checkout@v4
# - name: Install Rust
# run: rustup update $nightly && rustup default $nightly
# - name: Loom tests
# run: RUSTFLAGS="--cfg loom -Dwarnings" cargo test --test loom
# run: RUSTFLAGS="--cfg loom -Dwarnings" cargo test --lib

coverage:
name: cargo tarpaulin
Expand All @@ -215,8 +214,7 @@ jobs:
- stable
- nightly
- sanitizer
- valgrind
- miri
- valgrind
- minrust
steps:
- uses: actions/checkout@v3
Expand All @@ -243,7 +241,7 @@ jobs:
uses: actions-rs/cargo@v1
with:
command: tarpaulin
args: --run-types Tests,Doctests --workspace --out Xml
args: --all-features --run-types tests --run-types doctests --workspace --out xml
- name: Upload to codecov.io
uses: codecov/codecov-action@v3.1.1
with:
Expand Down
58 changes: 58 additions & 0 deletions .github/workflows/loc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: loc

on:
push:
branches:
- main
paths-ignore:
- 'README.md'
- 'COPYRIGHT'
- 'LICENSE*'
- '**.md'
- '**.txt'
- 'art'
pull_request:
paths-ignore:
- 'README.md'
- 'COPYRIGHT'
- 'LICENSE*'
- '**.md'
- '**.txt'
- 'art'
workflow_dispatch:

jobs:
loc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Install Rust
run: |
rustup update stable && rustup default stable
rustup component add clippy
rustup component add rustfmt
- name: Install tokeit
run: |
cargo install tokeit --force
- name: Count total lines of code
run: |
tokeit
- name: Upload total loc to GitHub Gist
uses: actions/github-script@v6
with:
github-token: ${{ secrets.GIST_PAT }}
script: |
const fs = require('fs');
const output = fs.readFileSync('tokeit.json', 'utf8');
const gistId = '327b2a8aef9003246e45c6e47fe63937';
await github.rest.gists.update({
gist_id: gistId,
files: {
"skl": {
content: output
}
}
});
44 changes: 30 additions & 14 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name = "skl"
version = "0.4.0"
edition = "2021"
repository = "https://github.com/al8n/skl-rs"
description = "A lock-free thread-safe concurrent ARENA based skiplist implementation which helps develop MVCC memtable for LSM-Tree. Inspired by Dgraph's badger https://github.com/dgraph-io/badger/tree/main/skl."
description = "A lock-free thread-safe concurrent ARENA based (heap backend or memory map backend) skiplist implementation which helps develop MVCC memtable for LSM-Tree."
documentation = "https://docs.rs/skl"
homepage = "https://github.com/al8n/skl-rs"
keywords = ["skiplist", "lock-free", "memtable", "concurrency", "arena"]
Expand All @@ -15,35 +15,48 @@ path = "benches/bench.rs"
name = "bench"
harness = false

[[test]]
path = "tests/loom.rs"
name = "loom"
[[example]]
name = "heap"
path = "examples/heap.rs"

[[example]]
name = "mmap"
path = "examples/mmap.rs"
required-features = ["memmap"]

[[example]]
name = "mmap-anon"
path = "examples/mmap_anon.rs"
required-features = ["memmap"]

[features]
default = ["std"]
alloc = []
mmap = ["memmapix", "fs4", "std"]
std = ["bytes/std", "rand/std", "rand/std_rng", "inline_more"]
inline_more = []
js = ["getrandom/js"]
memmap = ["memmap2", "fs4", "std"]
std = ["rand/default"]

[target.'cfg(loom)'.dependencies]
loom = "0.7"

[target.'cfg(target_family = "wasm")'.dependencies]
getrandom = { version = "0.2", features = ["js"] }
rand = { version = "0.8", default-features = false, features = ["getrandom"] }

[target.'cfg(not(target_family = "wasm"))'.dependencies]
rand = { version = "0.8", default-features = false }

[dependencies]
bytes = { version = "1.5", default-features = false }
crossbeam-utils = { version = "0.8", default-features = false }
fs4 = { version = "0.6", optional = true }
getrandom = { version = "0.2", optional = true }
memmapix = { version = "0.7", optional = true }
rand = { version = "0.8", default-features = false, features = ["getrandom"] }
viewit = "0.1.5"

fs4 = { version = "0.8", optional = true }
memmap2 = { version = "0.9", optional = true }

[dev-dependencies]
criterion = "0.5"
tempfile = "3"
parking_lot = "0.12"
paste = "1"
wg = { version = "0.9", default-features = false, features = ["std"] }

[profile.bench]
opt-level = 3
Expand All @@ -58,3 +71,6 @@ rpath = false
[package.metadata.docs.rs]
all-features = true
rustdoc-args = ["--cfg", "docsrs"]

[workspace]
members = ["integration"]
106 changes: 54 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,61 +4,51 @@

<div align="center">

[<img alt="github" src="https://img.shields.io/badge/github-skl--rs-8da0cb?style=for-the-badge&logo=Github" height="22">][Github-url]
[<img alt="github" src="https://img.shields.io/badge/github-al8n/skl-8da0cb?style=for-the-badge&logo=Github" height="22">][Github-url]
<img alt="LoC" src="https://img.shields.io/endpoint?url=https%3A%2F%2Fgist.githubusercontent.com%2Fal8n%2F327b2a8aef9003246e45c6e47fe63937%2Fraw%2Fskl" height="22">
[<img alt="Build" src="https://img.shields.io/github/actions/workflow/status/al8n/skl-rs/ci.yml?style=for-the-badge&logo=Github-Actions&label=CI" height="22">][CI-url]
[<img alt="codecov" src="https://img.shields.io/codecov/c/gh/al8n/skl-rs?style=for-the-badge&token=aek5JwyaAZ&logo=codecov" height="22">][codecov-url]

[<img alt="docs.rs" src="https://img.shields.io/badge/docs.rs-skl-66c2a5?style=for-the-badge&labelColor=555555&logo=data:image/svg+xml;base64,PHN2ZyByb2xlPSJpbWciIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmlld0JveD0iMCAwIDUxMiA1MTIiPjxwYXRoIGZpbGw9IiNmNWY1ZjUiIGQ9Ik00ODguNiAyNTAuMkwzOTIgMjE0VjEwNS41YzAtMTUtOS4zLTI4LjQtMjMuNC0zMy43bC0xMDAtMzcuNWMtOC4xLTMuMS0xNy4xLTMuMS0yNS4zIDBsLTEwMCAzNy41Yy0xNC4xIDUuMy0yMy40IDE4LjctMjMuNCAzMy43VjIxNGwtOTYuNiAzNi4yQzkuMyAyNTUuNSAwIDI2OC45IDAgMjgzLjlWMzk0YzAgMTMuNiA3LjcgMjYuMSAxOS45IDMyLjJsMTAwIDUwYzEwLjEgNS4xIDIyLjEgNS4xIDMyLjIgMGwxMDMuOS01MiAxMDMuOSA1MmMxMC4xIDUuMSAyMi4xIDUuMSAzMi4yIDBsMTAwLTUwYzEyLjItNi4xIDE5LjktMTguNiAxOS45LTMyLjJWMjgzLjljMC0xNS05LjMtMjguNC0yMy40LTMzLjd6TTM1OCAyMTQuOGwtODUgMzEuOXYtNjguMmw4NS0zN3Y3My4zek0xNTQgMTA0LjFsMTAyLTM4LjIgMTAyIDM4LjJ2LjZsLTEwMiA0MS40LTEwMi00MS40di0uNnptODQgMjkxLjFsLTg1IDQyLjV2LTc5LjFsODUtMzguOHY3NS40em0wLTExMmwtMTAyIDQxLjQtMTAyLTQxLjR2LS42bDEwMi0zOC4yIDEwMiAzOC4ydi42em0yNDAgMTEybC04NSA0Mi41di03OS4xbDg1LTM4Ljh2NzUuNHptMC0xMTJsLTEwMiA0MS40LTEwMi00MS40di0uNmwxMDItMzguMiAxMDIgMzguMnYuNnoiPjwvcGF0aD48L3N2Zz4K" height="20">][doc-url]
[<img alt="crates.io" src="https://img.shields.io/crates/v/skl?style=for-the-badge&logo=data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iaXNvLTg4NTktMSI/Pg0KPCEtLSBHZW5lcmF0b3I6IEFkb2JlIElsbHVzdHJhdG9yIDE5LjAuMCwgU1ZHIEV4cG9ydCBQbHVnLUluIC4gU1ZHIFZlcnNpb246IDYuMDAgQnVpbGQgMCkgIC0tPg0KPHN2ZyB2ZXJzaW9uPSIxLjEiIGlkPSJMYXllcl8xIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB4PSIwcHgiIHk9IjBweCINCgkgdmlld0JveD0iMCAwIDUxMiA1MTIiIHhtbDpzcGFjZT0icHJlc2VydmUiPg0KPGc+DQoJPGc+DQoJCTxwYXRoIGQ9Ik0yNTYsMEwzMS41MjgsMTEyLjIzNnYyODcuNTI4TDI1Niw1MTJsMjI0LjQ3Mi0xMTIuMjM2VjExMi4yMzZMMjU2LDB6IE0yMzQuMjc3LDQ1Mi41NjRMNzQuOTc0LDM3Mi45MTNWMTYwLjgxDQoJCQlsMTU5LjMwMyw3OS42NTFWNDUyLjU2NHogTTEwMS44MjYsMTI1LjY2MkwyNTYsNDguNTc2bDE1NC4xNzQsNzcuMDg3TDI1NiwyMDIuNzQ5TDEwMS44MjYsMTI1LjY2MnogTTQzNy4wMjYsMzcyLjkxMw0KCQkJbC0xNTkuMzAzLDc5LjY1MVYyNDAuNDYxbDE1OS4zMDMtNzkuNjUxVjM3Mi45MTN6IiBmaWxsPSIjRkZGIi8+DQoJPC9nPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPGc+DQo8L2c+DQo8Zz4NCjwvZz4NCjxnPg0KPC9nPg0KPC9zdmc+DQo=" height="22">][crates-url]
[<img alt="rustc" src="https://img.shields.io/badge/MSRV-1.72.0-fc8d62.svg?style=for-the-badge&logo=Rust" height="22">][rustc-url]
<img alt="license" src="https://img.shields.io/badge/License-Apache%202.0/MIT-blue.svg?style=for-the-badge&fontColor=white&logoColor=f5c076&logo=data:image/svg+xml;base64,PCFET0NUWVBFIHN2ZyBQVUJMSUMgIi0vL1czQy8vRFREIFNWRyAxLjEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4KDTwhLS0gVXBsb2FkZWQgdG86IFNWRyBSZXBvLCB3d3cuc3ZncmVwby5jb20sIFRyYW5zZm9ybWVkIGJ5OiBTVkcgUmVwbyBNaXhlciBUb29scyAtLT4KPHN2ZyBmaWxsPSIjZmZmZmZmIiBoZWlnaHQ9IjgwMHB4IiB3aWR0aD0iODAwcHgiIHZlcnNpb249IjEuMSIgaWQ9IkNhcGFfMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgdmlld0JveD0iMCAwIDI3Ni43MTUgMjc2LjcxNSIgeG1sOnNwYWNlPSJwcmVzZXJ2ZSIgc3Ryb2tlPSIjZmZmZmZmIj4KDTxnIGlkPSJTVkdSZXBvX2JnQ2FycmllciIgc3Ryb2tlLXdpZHRoPSIwIi8+Cg08ZyBpZD0iU1ZHUmVwb190cmFjZXJDYXJyaWVyIiBzdHJva2UtbGluZWNhcD0icm91bmQiIHN0cm9rZS1saW5lam9pbj0icm91bmQiLz4KDTxnIGlkPSJTVkdSZXBvX2ljb25DYXJyaWVyIj4gPGc+IDxwYXRoIGQ9Ik0xMzguMzU3LDBDNjIuMDY2LDAsMCw2Mi4wNjYsMCwxMzguMzU3czYyLjA2NiwxMzguMzU3LDEzOC4zNTcsMTM4LjM1N3MxMzguMzU3LTYyLjA2NiwxMzguMzU3LTEzOC4zNTcgUzIxNC42NDgsMCwxMzguMzU3LDB6IE0xMzguMzU3LDI1OC43MTVDNzEuOTkyLDI1OC43MTUsMTgsMjA0LjcyMywxOCwxMzguMzU3UzcxLjk5MiwxOCwxMzguMzU3LDE4IHMxMjAuMzU3LDUzLjk5MiwxMjAuMzU3LDEyMC4zNTdTMjA0LjcyMywyNTguNzE1LDEzOC4zNTcsMjU4LjcxNXoiLz4gPHBhdGggZD0iTTE5NC43OTgsMTYwLjkwM2MtNC4xODgtMi42NzctOS43NTMtMS40NTQtMTIuNDMyLDIuNzMyYy04LjY5NCwxMy41OTMtMjMuNTAzLDIxLjcwOC0zOS42MTQsMjEuNzA4IGMtMjUuOTA4LDAtNDYuOTg1LTIxLjA3OC00Ni45ODUtNDYuOTg2czIxLjA3Ny00Ni45ODYsNDYuOTg1LTQ2Ljk4NmMxNS42MzMsMCwzMC4yLDcuNzQ3LDM4Ljk2OCwyMC43MjMgYzIuNzgyLDQuMTE3LDguMzc1LDUuMjAxLDEyLjQ5NiwyLjQxOGM0LjExOC0yLjc4Miw1LjIwMS04LjM3NywyLjQxOC0xMi40OTZjLTEyLjExOC0xNy45MzctMzIuMjYyLTI4LjY0NS01My44ODItMjguNjQ1IGMtMzUuODMzLDAtNjQuOTg1LDI5LjE1Mi02NC45ODUsNjQuOTg2czI5LjE1Miw2NC45ODYsNjQuOTg1LDY0Ljk4NmMyMi4yODEsMCw0Mi43NTktMTEuMjE4LDU0Ljc3OC0zMC4wMDkgQzIwMC4yMDgsMTY5LjE0NywxOTguOTg1LDE2My41ODIsMTk0Ljc5OCwxNjAuOTAzeiIvPiA8L2c+IDwvZz4KDTwvc3ZnPg==" height="22">

[<img alt="license-apache" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg?style=for-the-badge&logo=Apache" height="22">][license-apache-url]
[<img alt="license-mit" src="https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge&fontColor=white&logoColor=f5c076&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGhlaWdodD0iMzZweCIgdmlld0JveD0iMCAwIDI0IDI0IiB3aWR0aD0iMzZweCIgZmlsbD0iI2Y1YzA3NiI+PHBhdGggZD0iTTAgMGgyNHYyNEgwVjB6IiBmaWxsPSJub25lIi8+PHBhdGggZD0iTTEwLjA4IDEwLjg2Yy4wNS0uMzMuMTYtLjYyLjMtLjg3cy4zNC0uNDYuNTktLjYyYy4yNC0uMTUuNTQtLjIyLjkxLS4yMy4yMy4wMS40NC4wNS42My4xMy4yLjA5LjM4LjIxLjUyLjM2cy4yNS4zMy4zNC41My4xMy40Mi4xNC42NGgxLjc5Yy0uMDItLjQ3LS4xMS0uOS0uMjgtMS4yOXMtLjQtLjczLS43LTEuMDEtLjY2LS41LTEuMDgtLjY2LS44OC0uMjMtMS4zOS0uMjNjLS42NSAwLTEuMjIuMTEtMS43LjM0cy0uODguNTMtMS4yLjkyLS41Ni44NC0uNzEgMS4zNlM4IDExLjI5IDggMTEuODd2LjI3YzAgLjU4LjA4IDEuMTIuMjMgMS42NHMuMzkuOTcuNzEgMS4zNS43Mi42OSAxLjIuOTFjLjQ4LjIyIDEuMDUuMzQgMS43LjM0LjQ3IDAgLjkxLS4wOCAxLjMyLS4yM3MuNzctLjM2IDEuMDgtLjYzLjU2LS41OC43NC0uOTQuMjktLjc0LjMtMS4xNWgtMS43OWMtLjAxLjIxLS4wNi40LS4xNS41OHMtLjIxLjMzLS4zNi40Ni0uMzIuMjMtLjUyLjNjLS4xOS4wNy0uMzkuMDktLjYuMS0uMzYtLjAxLS42Ni0uMDgtLjg5LS4yMy0uMjUtLjE2LS40NS0uMzctLjU5LS42MnMtLjI1LS41NS0uMy0uODgtLjA4LS42Ny0uMDgtMXYtLjI3YzAtLjM1LjAzLS42OC4wOC0xLjAxek0xMiAyQzYuNDggMiAyIDYuNDggMiAxMnM0LjQ4IDEwIDEwIDEwIDEwLTQuNDggMTAtMTBTMTcuNTIgMiAxMiAyem0wIDE4Yy00LjQxIDAtOC0zLjU5LTgtOHMzLjU5LTggOC04IDggMy41OSA4IDgtMy41OSA4LTggOHoiLz48L3N2Zz4=" height="22">][license-mit-url]

A lock-free thread-safe concurrent ARENA based skiplist implementation which helps develop MVCC memtable for LSM-Tree. Inspired by [Dgraph's badger](https://github.com/dgraph-io/badger/tree/main/skl).
1. A lock-free thread-safe concurrent `SkipMap` and `SkipSet` implementation based on ARENA skiplist which helps develop MVCC memtable for LSM-Tree.
2. A lock-free thread-safe concurrent memory map based on-disk `SkipMap` and `SkipSet`.

</div>

## Installation

```toml
[dependencies]
skl = "0.3"
```

## Example

```rust
use skl::SkipMap;
use std::sync::Arc;

fn main() {
const N: usize = 1000;
let l = Arc::new(SkipMap::new(1 << 20));
let wg = Arc::new(());
for i in 0..N {
let w = wg.clone();
let l = l.clone();
std::thread::spawn(move || {
l.insert(key(i), new_value(i));
drop(w);
});
}
while Arc::strong_count(&wg) > 1 {}
for i in 0..N {
let w = wg.clone();
let l = l.clone();
std::thread::spawn(move || {
assert_eq!(
l.get(key(i)).unwrap().as_value_ref(),
new_value(i).as_value_ref(),
"broken: {i}"
);
drop(w);
});
}
}
```
- Only use heap backend (suppport `no_std`)

```toml
[dependencies]
skl = "0.4"
```

- Enable memory map backend

```toml
[dependencies]
skl = { version = "0.4", features = ["memmap"] }
```

## Features

- **Lock-free and Concurrent-Safe:** SkipMap and SkipSet provide lock-free operations, ensuring efficient concurrent access without the need for explicit locking mechanisms.
- **Extensible for Key-Value Database Developers:** Designed as a low-level crate, SkipMap and SkipSet offer a flexible foundation for key-value database developers. You can easily build your own memtable or write-ahead-log (WAL) using these structures.
- **Memory Efficiency:** These data structures are optimized for minimal memory overhead. They operate around references, avoiding unnecessary allocations and deep copies, which can be crucial for efficient memory usage.
- **Efficient Iteration:** Enjoy fast forward and backward iteration through the elements in your SkipMap or SkipSet. Additionally, bounded iterators are supported, allowing you to traverse only a specified range of elements efficiently.
- **Snapshot Support:** Create snapshots of your SkipMap or SkipSet, offering a read-only view of the contents at a specific moment in time. Snapshots provide a consistent view of the data, enabling implementations of transactional semantics and other use cases where data consistency is crucial.
- **Memory Management Options:**
- **Heap Allocation:** Memory allocation is handled by Rust's allocator, ensuring all data resides in RAM.
- **Mmap:** Data can be mapped to a disk file by the operating system, making it suitable for write-ahead-logs (WAL) and durable storage.
- **Mmap Anonymous:** Mapped to anonymous memory (virtual memory) by the OS, ideal for large in-memory memtables, optimizing memory utilization.

## Examples

Please see [examples](https://github.com/al8n/skl-rs/tree/main/examples) folder for more details.

## Tests

Expand Down Expand Up @@ -93,17 +83,29 @@ fn main() {
| x86_64-pc-windows-gnu | &#9989; |
| x86_64-linux-android | &#9989; |

## Pedigree

This code is inspired and modified based on Cockroachdb's pebble arenaskl code:

https://github.com/cockroachdb/pebble/tree/master/internal/arenaskl

The pebble's arenaskl code is based on Andy Kimball's arenaskl code:

https://github.com/andy-kimball/arenaskl

The arenaskl code is based on the skiplist found in Badger, a Go-based
KV store:

https://github.com/dgraph-io/badger/tree/master/skl

The skiplist in Badger is itself based on a C++ skiplist built for
Facebook's RocksDB:

https://github.com/facebook/rocksdb/tree/master/memtable

## TODO (help wanted)

- [x] make the crate test cases pass `cargo miri`
- [ ] make the crate test cases pass `cargo loom`
- [ ] Implement
- [ ] `std::iter::Iterator`
- [ ] `get_or_insert`
- [ ] `remove`
- [ ] `contains`
- [ ] change signature from `insert(k, v)` => `insert(k, v) -> Option<ValueRef>`
- [ ] mmap backend (currently is vector backend)

#### License

Expand Down
Loading

0 comments on commit 37ceb55

Please sign in to comment.