Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to query minimum set of ingesters #5202

Merged
merged 4 commits into from
Jun 9, 2023

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Jun 8, 2023

What this PR does

This PR adds experimental support for querying the minimal set of ingesters required to reach quorum.

With -querier.minimize-ingester-requests=false (the default), queriers behave as they do today: they send a request to all ingesters in the tenant's shard, wait until quorum is reached and then cancel any outstanding unnecessary requests.

With -querier.minimize-ingester-requests=true, queriers instead initially only send requests to the minimum number of ingesters required to reach quorum, and initiate requests to further ingesters if any of the initial requests fail. (If enough requests fail that quorum is impossible, all requests are cancelled as before, and no further requests are made.)

This reduces querier and ingester resource utilisation on the happy path, at the cost of increased latency on the unhappy path where one or more ingesters return an error, fail to respond, or respond slowly.

Which issue(s) this PR fixes or relates to

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@charleskorn charleskorn force-pushed the charleskorn/query-2-of-3-ingesters branch from 3139ce8 to e049bba Compare June 9, 2023 00:43
@charleskorn charleskorn marked this pull request as ready for review June 9, 2023 00:44
@charleskorn charleskorn requested review from a team as code owners June 9, 2023 00:44
@charleskorn charleskorn enabled auto-merge (squash) June 9, 2023 00:44
@charleskorn charleskorn merged commit 70a1cfc into main Jun 9, 2023
@charleskorn charleskorn deleted the charleskorn/query-2-of-3-ingesters branch June 9, 2023 00:57
@charleskorn charleskorn mentioned this pull request Jun 26, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants