-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Append to small bz1 blocks #3735
Conversation
@benbjohnson -- rebase your changes to pick up the test I have skipped. |
5313724
to
06eb4b9
Compare
@otoolep I rebased and updated the |
Is this an optimization? It that its purpose? |
@otoolep flushes can happen from the WAL due to memory pressure. The goal of this is to ensure that compressed blocks are big enough to actually get value from the compression. If you compress a single data point, your ratio won't be nearly as good as when you compress 1,000. Also, lumping these together lessons the number of unique keys in the DB. |
I see, a fix, makes sense @pauldix |
@benbjohnson does the quick test cover all three scenarios? If so, +1 |
@pauldix It should but I'll add a more explicit append test for it. |
+1 once that test is in. Also good to get this cherry picked into the 0.9.3 branch. |
207b058
to
f7ae6f8
Compare
This commit changes the bz1 append to check for a small ending block first. If the block is below the threshold for block size then it is rewritten with the new data points instead of having a new block written.
f7ae6f8
to
e57d602
Compare
Append to small bz1 blocks
Overview
This pull request changes the
bz1
append to check for a small ending block first. If the block is below the threshold for block size then it is rewritten with the new data points instead of having a new block written./cc @pauldix