Tombstone memory improvements #7084
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I only took a look at the query engine portion of this, but that part's LGTM. |
| } | ||
|
|
||
| resC := make(chan res) | ||
| var n int |
There was a problem hiding this comment.
nit: this is just a style thing, but since you know you're sending a result for each file, couldn't you get rid of n completely and just create a buffered channel of length len(f.files)?
In your gathering loop at the bottom of walkFiles, you would then loop len(resC) times.
|
Just a few nits, otherwise LGTM 👍 |
| // struct to hold the result of opening each reader in a goroutine | ||
| type res struct { | ||
| err error | ||
| } |
There was a problem hiding this comment.
Seems like you could just do a chan error since there's only one field in the struct.
|
I agree with @e-dard's nits but otherwise LGTM. |
| } | ||
| batch = append(batch, ts.Key) | ||
|
|
||
| if len(batch) > 4096 { |
There was a problem hiding this comment.
len(batch) > 4096 means the original capacity of batch was exceeded and the slice had to grow. Nit, but should this be >= 4096?
Tombstone were read fully into memory at startup which could consume a lot of RAM and OOM the process if there were a lot of deleted series and many TSM files. This now walks the tombstone file and iteratively applies the tombstone which uses significantly less RAM. This may be slightly slower in the generate cause, but should scale better.
Use a bufio.Scanner to read v1 tombstones instead of reading in the whole file and parsing it from memory.
This keeps some memory bounds when reloading a TSM files tombstones so that the heap does not grow exceedintly fast and stay there after the deletes are applied.
If there were multiple TSM files and a delete/drop was run, we would write the delete series to the tombstone file N times for each file. This occurred because FileStore.WalkKeys walks every key in every TSM file which can return duplicate keys. This issue caused TSM files to be much larger than they should be and also cause large memory usage during the delete.
Aux and condition iterators where not closed which could cause TSM files to leak if they were queried against while a compaction was running.
If they were left around, re-enabling them again could cause future compactions to continuously fail. A restart of the server would clean them up correctly though.
break cause the first one to be tracked and all others would leak as temp files that would not be removed until the server restarted.
The path info only contained the file name which caused tombstone files to not be removed if there were queries running against a file that was compacted. This is now consistent with the TSMReader.Path which returns the full path info.
|
👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for all non-trivial PRs
This PR has a number of improvements to reduce memory usage, improve performance and fix bugs related to tombstone files.