What is the best way to bulk load redb?

[marvin-j97]

I am trying to benchmark large data sets in https://github.com/marvin-j97/rust-storage-bench, so I want to load a lot of data very quickly, no matter the durability.

If I use Durability::None, that bloats the disk size, which already caused me to run of disk space more than once, so I started doing an Immediate flush here and there:

// Note the keys are written in monotonically increasing order.
for x in 0..item_count {
    db.insert(
        key,
        value,
        // NOTE: Avoid too much disk space building up...
        args.backend == Backend::Redb && x % 100_000 == 0,
    );
}

With the above code, writing 100M 16 byte keys and 64 byte values takes 130 minutes, which is about 78µs per insert, which is slower than the fsync time of my SSD (pm9a3), so it looks like there is no advantage to write with Durability=None. So is there any point in using None at all?

Additionally, for this comparatively small data set (it's just ~8 GB of user data), redb has written 4.4 TB (write amp = 540), with the resulting .redb file being ~28GB.

As a comparison

• sled 0.x takes 5 minutes, and comparable disk space
• fjall takes 3 minutes, and uses 8 GB (to be expected, because LSM)

What is the best way to write a lot of KVs without bloating disk space, while keeping inserts somewhat fast?

[cberner]

It's best to insert them all in a single transaction, and then the durability won't matter. Here's an example:

redb/benches/lmdb_benchmark.rs

Lines 79 to 88 in 7377b3b

	let mut txn = db.write_transaction();
	let mut inserter = txn.get_inserter();
	{
	for _ in 0..ELEMENTS {
	let (key, value) = gen_pair(&mut rng);
	inserter.insert(&key, &value).unwrap();
	}
	}
	drop(inserter);
	txn.commit().unwrap();

If you're able to insert them in sorted order, that might improve write amplification. Alternately, you can adjust the cache size, if you have enough RAM

[marvin-j97]

That works better, down to ~2.74µs per item