Reduce disk space with RocksDB compaction and compression

[teor2345]

Is your feature request related to a problem? Please describe.

RocksDB logs every write to a MANIFEST file, then replays them on load. Similarly, the write-ahead log can grow really large.
We should limit the size of the manifest file and write-ahead log file.
https://github.com/facebook/rocksdb/wiki/Speed-Up-DB-Open#manifest-replay

RocksDB does single-threaded compaction by default, but also supports multi-threaded compaction.
https://github.com/facebook/rocksdb/wiki/Compaction

RocksDB can also open files with multiple threads:
https://github.com/facebook/rocksdb/wiki/Speed-Up-DB-Open#reading-footer-and-meta-blocks-of-all-the-sst-files

RocksDB supports block-level (database page) compression:
https://github.com/facebook/rocksdb/wiki/Compression
https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format

Some of our data is incompressible (hashes, keys, proofs), but other data is compressible (heights, empty fields, duplicated data).

If the data compresses well, we'll save disk space. And if the machine has spare CPU, we'll see a speedup loading data from disk, too.

Describe the solution you'd like

Limit the size of the manifest file and write-ahead log file:

• options.max_manifest_file_size
• options.max_total_wal_size

Multithreaded compaction:
Set options.max_background_compactions to 4.
https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#other-general-options

Set options.max_file_opening_threads to 2.
https://github.com/facebook/rocksdb/wiki/Speed-Up-DB-Open#reading-footer-and-meta-blocks-of-all-the-sst-files

Lightweight compression:
Set RocksDB's options.compression to LZ4, and compare the database size from a new load.

Heavyweight compression:
Also try setting options.bottommost_compression to ZSTD, and do a comparison with lightweight and no compression.

Per-Column Family compression:
Enable compression only for column families containing Height or Block (and other data which contains runs of zeroes, or repeated patterns).
https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ#basic-readwrite

Describe alternatives you've considered

Do nothing.

Compression isn't really a high priority.

Alternate compaction:
Set options.compaction_style to kCompactionStyleUniversal, and compare the database size from a new load.

Universal compaction has size limits and a double-size issue, so we don't want to use it:
https://github.com/facebook/rocksdb/wiki/Universal-Compaction#double-size-issue

[hdevalence]

I don't think that compression will benefit us much, because most of our data (by weight) is already cryptographically random, and the small amount of data that isn't is largely inherited from Bitcoin, which already uses a custom space-efficient format.

On the other hand, compaction seems relatively more useful.

[conradoplg]

Since we store trees indexed by the current tip height (and therefore delete the previous and create a new entry for every block), it's making the state big since compaction doesn't seem to be triggered automatically (it is done when the node starts, which I confirmed, and reduces the state size considerably)

[teor2345]

This hasn't been a problem so far, we can re-open if it becomes one.

We should consider this change if it is needed for lightwalletd performance.