Add option for smaller bitcask keys #643
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The kv bucket/key to bitcask key conversion uses term_to_binary. This results in 13 bytes of overhead per key.
This patch introduces a new key format:
The first two bytes specify the length of the bucket, allowing kv to translate the compact binary back to the bucket/key pair that it needs. In order to read old-format values off of disk, this code allows the insertion of a translation function, so that keys in the keydir are in a consistent format. This saves 11 bytes per entry in the keydir.
When a large percentage of your keys on disk are in the old format, there is a small but noticable slowdown in startup time and fold time. This should go down over time as keys are replaced. If the slowdown is intolerable, adding
{small_keys, false}
to the bitcask section of your app.config will revert to the old format. This feature is currently on by default; whether this should be the case is up for debate.This PR depends on a change in bitcask adding support for key transformation functions:
basho/bitcask#103