Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazily create merge files to avoid creation of empty files #115

Merged
merged 1 commit into from
Nov 7, 2013

Conversation

evanmcc
Copy link
Contributor

@evanmcc evanmcc commented Oct 28, 2013

in situations with few hot keys, you can produce a whole ton of empty data files. when all of keys in a partial merge have already expired, a data file will be created and nothing will ever be put into it. These files aren't cleaned up until reopen, so they can cause problems when may of them accumulate over time.

Moving to a lazy file creation strategy avoids this problem.

fixes #113

Testing notes:

See #116 for test setup.

The symptom here is a buildup of 0 length data files paired with empty 18 byte hint files. These should be readily apparent in the setup given above, they will begin to appear as soon as merges start, and will build up linearly over time. On an overnight run, tens of thousands can build up.

None should appear (except very briefly, as new files are started) with this patch applied.

@evanmcc
Copy link
Contributor Author

evanmcc commented Oct 28, 2013

Have done a little testing on this now. This does seem to avoid the tight put loop on small keysets case. Passes unit tests.

@evanmcc evanmcc mentioned this pull request Nov 6, 2013
@slfritchie
Copy link
Contributor

Hi, Evan. The code looks good, and all the tests (including the PULSE test) are happy. NOTE that the bitcask_qc_expiry test requires a commit that I put onto 'develop' earlier today. +1, thanks!

evanmcc added a commit that referenced this pull request Nov 7, 2013
lazily create merge files to avoid creation of empty files
@evanmcc evanmcc merged commit a1fd61d into develop Nov 7, 2013
evanmcc added a commit that referenced this pull request Nov 7, 2013
doing this to bring the branch up to date and fix a known merge
issue w/ PR #115
@evanmcc evanmcc mentioned this pull request Nov 7, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

File descriptor leak + zero byte file leak on aggressive merge activity
2 participants