lazily create merge files to avoid creation of empty files #115
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
in situations with few hot keys, you can produce a whole ton of empty data files. when all of keys in a partial merge have already expired, a data file will be created and nothing will ever be put into it. These files aren't cleaned up until reopen, so they can cause problems when may of them accumulate over time.
Moving to a lazy file creation strategy avoids this problem.
fixes #113
Testing notes:
See #116 for test setup.
The symptom here is a buildup of 0 length data files paired with empty 18 byte hint files. These should be readily apparent in the setup given above, they will begin to appear as soon as merges start, and will build up linearly over time. On an overnight run, tens of thousands can build up.
None should appear (except very briefly, as new files are started) with this patch applied.