Fix/Rocksdb cache not set

[asdacap]

• Originally, I thought of making the shared cache that we set across rocksdb table explicit.
• But then it turns out because the cache is setup after the option is setup, in effect no block cache was set, causing it all rocksdb table to use 32 MB (or 8 MB, I don't know depends on rocksdb version) of block cache. For reference, the statedb's block cache is suppose to be about 900MB.
• But then, if I set the block cache properly, block processing time becomes more spiky. Cache hit metric increase, less IO is observed, but block processing throughput suffer. In fact less cache for state improve block processing time stability.
• I suspected that the block size (16K) is too large, causing each node lookup to have to fetch 16K in order to populate the cache.
• True enough, setting a much lower block size (1K) significantly reduce IO (in terms of bps) by nearly 10x. However, lower block size comes with the downsize of proportionally higher number of block index (higher memory consumption), and higher IOPs.
• To compensate for that, TwoLevelIndex is turned on, which significanly reduce the in-memory part of the index. CacheIndexAndFilterBlocks is also advised, which will cause the block index to fit in the block cache. With an existing DB, if CacheIndexAndFilterBlocks is set to true, memory usage decrease by about 1GB at start, but performance suffer, likely due to larger index and block size, making the cache less effective. CacheIndexAndFilterBlocks remains turned off, but a fresh sync db can turn that on to reduce memory.
• Lower block size also comes with the downsize of higher iops. Experiment shows the different in block processing time between 4K and 512 block size is negligable. IO in terms of bps reduce even further with 512, but in terms of ops increase a little bit. Index cache hit rate goes down (512 would have 8 times more index than 4k), and data cache hit goes down too, making it less effective than 4k. Lower block size also increase the db size a little bit (about 10GB for 512 compared to 4K).
• Anyway, I've set it to 4K by default fo state db and make it configurable.
• Long story short, improved block processing time and consistency and reduced memory usage.
• Require a resync/full prune to take effect. Or wait a few days for whole db to compact/rebuild.

Testing

Tested via running trace block of past 2000 blocks in sequence. In memory pruning is disabled. Single thread only.

6 runs in sequence:

• Old config
• Old config with cache properly setup
• Old config with CacheIndexAndFilterBlocks and cache?
• 4K block size with block hash index type (some point lookup optimization that does not seems to do anything). CacheIndexAndFilterBlocks on.
• 4K block size. CacheIndexAndFilterBlocks on.
• 512 block size. CacheIndexAndFilterBlocks on.

• Lower block size consistently shows lower IO (in bps) and higher throughput (request per sec).
• IOPs never reach above 50k ops except with 512 block size.
• Something is limiting my traceblock to 8 request per second. I've no idea what.

Changes

• Explicitly specify shared block cache.
• Copy OptimizeForPointLookup code to take effect.
• Allow specifying separate block cache per table.
• Allow skipping memory hint setting completely.
• Set block size for state db to 4K.
• Enable CacheIndexAndFilterBlocks and two level index.

Types of changes

What types of changes does your code introduce?

• Cleanup
• Refactoring
• Optimization

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

[asdacap]

Turns out, currently because InitCache happens after BuildOptions, the passed in _cache is 0, meaning they are probably are using the default 8MB block cache per table, which explain the strangely much higher cache hit.

Before -> After

Much more unstable block processing though. Gonna try to separate state cache from other cache.

[asdacap]

Confusingly, lower block cache equals faster block processing time?

[asdacap]

Seems to be caused by the block size, which could be too big, so the work populating the cache is probably bottlenecking the fetch. Reducing the block size to 1024, then setting up partitioned index shows 8 to 10 times less IO (in terms of bytes per sec), but increase iops. Uses less memory it seems, so thats nice too.

Graph is 7GB cache 1K block, 1GB cache 1K block, forward syncing a backup 16K block, 16K 8MB cache, 16K 1GB cache. All runs traceblock for past 500 block.

[benaadams]

Test errors look actual

Failed Cache_state_index("archive",False) [128 ms]
Error Message:
Expected propertyValue to be False because ropsten_archive.cfg: IDbConfig.CacheIndexAndFilterBlocks, but found True.

Failed Caches_in_fast_blocks("fast") [1 ms]
Error Message:
Expected propertyValue to be False because ropsten.cfg: IDbConfig.HeadersDbCacheIndexAndFilterBlocks, but found True.

Failed Cache_state_index("^archive",False) [145 ms]
Error Message:
Expected propertyValue to be False because ropsten.cfg: IDbConfig.CacheIndexAndFilterBlocks, but found True.

[asdacap]

Yes I'm turning that on with two level index type. Seems to significantly reduce memory as advertised. Does not seems to reduce memory with standard index, though.

[asdacap]

Actually, it does reduce memory, but massively reduce performance, likely due to very low block cache which was not configured correctly.

[benaadams]

Is a merge conflict

[asdacap]

For reference. 80% data cache hit, improved processing time, but not much. Significantly improved 50p for state get. Significantly lower IO and iops.

(after, before)

[asdacap]

I made a mistake, the above result is with caching store disabled. Not sure about if caching store enabled.