Handle distributed queries when shards != data nodes #2327
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #2272 There was previously a explict panic put in the query engine to prevent queries where the number of shards was not equal to the number of data nodes in the cluster. This was waiting for the distributed queries branch to land but was not removed when that landed. There may be a more efficient way to do fix this but this fix simply queries all the shards and merges their outputs. Previously, the code assumed that only one shard would be hit. Querying multiple shards ended up producing duplicate values during the map phase so the map output needed to be merged as opposed to appended to avoid the dups.
| @@ -204,6 +204,50 @@ func (m *MapReduceJob) Execute(out chan *Row, filterEmptyResults bool) { | |||
| out <- row | |||
| } | |||
|
|
|||
| // mergeOutputs merges two sorted slices of rawQueryMapOutput such that duplicate | |||
There was a problem hiding this comment.
I don't follow -- how can we end up with duplicate timestamps? Data should only be de-duped if the series and timestamp is the same. But for a given series we should only be hitting 1 shard. I don't see how dupes can arise.
There was a problem hiding this comment.
It's happens from removing the hard coding of sg.shards[0]here: https://github.com/influxdb/influxdb/blob/master/tx.go#L147
It's now looping over each shard.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2272
There was previously an explict panic put in the query engine to prevent
queries where the number of shards was not equal to the number of data nodes
in the cluster. This was waiting for the distributed queries branch to land
but was not removed when that landed.
There may be a more efficient way to do fix this but this fix simply queries
all the shards and merges their outputs. Previously, the code assumed that
only one shard would be hit. Querying multiple shards ended up producing
duplicate values during the map phase so the map output needed to be merged
as opposed to appended to avoid the dups.