Fix race in WALEntry.Encode and Values.Deduplicate #8095
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jwilder
changed the title
Under high query load, a race exists in the cache and the WAL. Since writes currently hit the cache first, they are availble for query before they hit the WAL. If the WAL is writing and accessign the Value slice at the same time that a query is run that needs to dedup the same slice, a race occurs. To fix this, the cache now just copies the values instead of storing the slice passed in. Another way to fix this might be to have the writes go to the wal before the cache. I think the latter would be better, but it introduces some larger write path issues that we'd need to also address. e.g. if the cache was full, writes to the WAL would need to be rejected to avoid filling the disk. Copying the slice in the cache is simpler for now and does not appear to dramatically affect performance.
There is a race where the field type can be deleted while a new type is written and during a query. When this happens, an iterator for the new type is created but old data make still exist in the cache for TSM files causing a panic.
benbjohnson
approved these changes
Mar 6, 2017
e-dard
approved these changes
Mar 6, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Under high query load, a race exists in the cache and the WAL. Since
writes currently hit the cache first, they are availble for query before
they hit the WAL. If the WAL is writing and accessign the Value slice
at the same time that a query is run that needs to dedup the same slice,
a race occurs.
To fix this, the cache now just copies the values instead of storing the
slice passed in. Another way to fix this might be to have the writes go
to the wal before the cache. I think the latter would be better, but it
introduces some larger write path issues that we'd need to also address.
e.g. if the cache was full, writes to the WAL would need to be rejected
to avoid filling the disk.
Copying the slice in the cache is simpler for now and does not appear to
dramatically affect performance.
Required for all non-trivial PRs
Required only if applicable
You can erase any checkboxes below this note if they are not applicable to your Pull Request.