Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split block data into dedicated RocksDB blob files #3138

Closed
teor2345 opened this issue Dec 3, 2021 · 1 comment
Closed

Split block data into dedicated RocksDB blob files #3138

teor2345 opened this issue Dec 3, 2021 · 1 comment
Labels
A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement I-slow Problems with performance or responsiveness S-needs-investigation Status: Needs further investigation

Comments

@teor2345
Copy link
Contributor

teor2345 commented Dec 3, 2021

Motivation

We can increase Zebra's speed by splitting RocksDB block and transaction data into separate blob files.

Specifications

The new BlobDB can be configured (on a per-column family basis if needed) simply by using the following options:
enable_blob_files: set it to true to enable key-value separation.
...

https://github.com/facebook/rocksdb/wiki/BlobDB#api

Rust API

The current version of the rocksdb crate does not have Rust bindings for the blob options:
https://docs.rs/rocksdb/0.17.0/rocksdb/struct.Options.html

So we would have to use the raw option API:
https://docs.rs/rocksdb/0.17.0/rocksdb/struct.DBWithThreadMode.html#method.set_options_cf

Related Work

We might need to do this as part of lightwalletd support, if the performance isn't good enough. See #3134.

@teor2345 teor2345 added A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement S-needs-triage Status: A bug report needs triage S-needs-investigation Status: Needs further investigation I-slow Problems with performance or responsiveness P-Optional labels Dec 3, 2021
@teor2345
Copy link
Contributor Author

teor2345 commented Mar 1, 2022

We only need to do this if performance is unacceptable.

This could be a cause of:

@teor2345 teor2345 closed this as completed Mar 1, 2022
@mpguerra mpguerra removed the S-needs-triage Status: A bug report needs triage label Sep 28, 2022
@teor2345 teor2345 reopened this Oct 19, 2022
@teor2345 teor2345 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement I-slow Problems with performance or responsiveness S-needs-investigation Status: Needs further investigation
Projects
None yet
Development

No branches or pull requests

2 participants