Add prefix bloom filter support #151

zaidoon1 · 2025-09-01T16:14:59Z

resolve #97

zaidoon1 · 2025-09-11T17:34:01Z

@marvin-j97 just want to make sure, did you want me to address the linter failures? they seem to be related to existing code

marvin-j97 · 2025-09-11T17:50:20Z

@marvin-j97 just want to make sure, did you want me to address the linter failures? they seem to be related to existing code

Linting seems to work now on the 3.0.0 branch, if you resolve the merge conflicts and pull, it should be fine.

zaidoon1 · 2025-09-11T17:59:39Z

hmm: help: use'_for type paths | 298 | pub(crate) fn get_binary_index_reader(&self) -> BinaryIndexReader<'_> { | ++++

hmm clippy is still not happy but i haven't changed this stuff? am i basing this off the right branch?

marvin-j97 · 2025-09-11T18:05:04Z

hmm: help: use'_for type paths | 298 | pub(crate) fn get_binary_index_reader(&self) -> BinaryIndexReader<'_> { | ++++
hmm clippy is still not happy but i haven't changed this stuff? am i basing this off the right branch?

That's just a warning though, right? I haven't really gotten to running clippy lints over the entire code base, so there are a lot of warnings as of now.

zaidoon1 · 2025-09-11T18:06:43Z

going to take a closer look, also noticed some api's have changed so my tests are failing now, i'll fix that and test locally before pushing

zaidoon1 · 2025-09-11T18:36:47Z

ok it was my fault, it was hidden with the other clippy warnings

src/segment/mod.rs

src/segment/writer/mod.rs

src/run_reader.rs

src/segment/filter/blocked_bloom/mod.rs

src/metrics.rs

src/prefix.rs

src/segment/util.rs

src/segment/mod.rs

src/run_reader.rs

src/config.rs

src/run_reader.rs

zaidoon1 · 2025-09-14T16:54:08Z

did i break the build and test lsm example build or some other isssue?

marvin-j97 · 2025-09-14T16:56:44Z

did i break the build and test lsm example build or some other isssue?

The CI examples script does not support raw .rs files in the examples/ folder, so it crashes.

zaidoon1 · 2025-09-14T17:51:10Z

did i break the build and test lsm example build or some other isssue?

The CI examples script does not support raw .rs files in the examples/ folder, so it crashes.

should be fixed now, thanks!

marvin-j97 · 2025-09-18T21:34:28Z

src/prefix.rs

+    /// Returns a unique name for this prefix extractor.
+    fn name(&self) -> &str;


I'm thinking how to make use of this.
I guess this should be stored in each keyspace's config (in fjall) to check that the given prefix extractor matches the one that was initially configured.

right, I may need to do more work but in rocksdb, if you want to change your prefix extractor then you need to use a different name which would cause rocksdb to ignore the old bloom that was used for the existing files

Then maybe the prefix extractor name also needs to be stored in every table's properties.

I've pushed a commit to do this and match rocksdb's behaviour:

"If prefix extractor changes when DB restarts, some SST files may contain prefix bloom filters generated using different prefix extractor than the one in current options. When opening those files, RocksDB will compare prefix extractor name stored in the properties of the SST files. If the name is different from the prefix extractor provided in the options, the prefix bloom filter is not used for this file"

one thing is how does fjall handle options that can be dynamically changeable vs can't be changed without a db restart? I don't think we would want a user to be able to change a prefix extractor on the fly

Currently the configuration cannot be changed during runtime. However I don't think it would be a problem, because tables that use the old prefix extractor would continue using that, and otherwise the filter is skipped. But that's probably an edge case and just shouldn't be allowed.

ok cool, so then what I have here should be good.

marvin-j97 added enhancement New feature or request performance epic api type:filters labels Sep 1, 2025

zaidoon1 force-pushed the prefix-bloom-3.0.0 branch from e27b31c to 9d8b4f0 Compare September 11, 2025 17:53

zaidoon1 force-pushed the prefix-bloom-3.0.0 branch 2 times, most recently from 5c502c3 to 7f70eeb Compare September 11, 2025 18:36

marvin-j97 requested changes Sep 12, 2025

View reviewed changes

src/segment/mod.rs Outdated Show resolved Hide resolved

src/segment/mod.rs Outdated Show resolved Hide resolved

zaidoon1 force-pushed the prefix-bloom-3.0.0 branch from cc303e0 to 8b995d4 Compare September 12, 2025 23:04

zaidoon1 requested a review from marvin-j97 September 12, 2025 23:07

zaidoon1 force-pushed the prefix-bloom-3.0.0 branch 3 times, most recently from 9a4719b to 5e53adb Compare September 13, 2025 08:06