-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add optional ring support to overrides-exporter #3908
Conversation
b6ec9de
to
65ee964
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job!
9383723
to
c03b2b6
Compare
Co-authored-by: Marco Pracucci <marco@pracucci.com>
e2e8e20
to
cccf038
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job! Few last comments. I just realised we may need the wait ring stability. Let's talk on Slack.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks to address my feedback, LGTM! We discussed offline about improvements to reduce the chances tenants will be resharded among replicas during a rollout, and we'll address them in a follow up PR.
* Use assert statements instead of handmade checks (#3936) jsonnet already has an 'assert <assertion>' statement, there's no need to craft handmade 'if <assertion> then null else error'. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Fixes templating parsing for manually created vars (#3921) When you create a dashboard variable from within the Grafana UI, it creates the query field as an object. The object contains a refId as well as the original query. The existing code expects the query to be a string, not an object and will therefor miss any metrics used if the variable was created in the UI. * Add optional ring support to overrides-exporter (#3908) * Add optional ring support to overrides-exporter * add docs * add config for read-write mode * fix import * fix naming, swallow error on ring failure * make format * make doc * add license * inline ownership check * add memberlist dependency for overrides exporter * basic integration test * [wip] troubleshoot ring membership * start ring client * fix instance address * Move code * wait on active instance state at startup * fix copy-pasta * improve integration tests * refactor * formatting * inline * move code around * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> * incorporate review feedback * basic unit test * mock ring in unit test * format * clean up * cosmetics/changelog * cosmetic * cosmetic * changelog/log * improve waiting in tests * use GetAddresses() Co-authored-by: Marco Pracucci <marco@pracucci.com> * Update test Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Jack <57678801+mothershipper@users.noreply.github.com> Co-authored-by: Felix Beuke <felix.j.beuke@gmail.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>
What this PR does
In read-write deployment mode, every backend replica includes an overrides-exporter that exports per-tenant limit metrics, leading to duplicate metrics being exposed if the number of replicas is greater than one. This PR adds experimental support for a ring to overrides-exporter, which is used to uniquely shard tenant metrics to a given instance and avoid the export of duplicate metrics.
While the solution proposed in this PR works for periods of stable service operations, it is still prone to duplicate exports during rollouts because of the effects of tenant resharding. This problem will be addressed in a follow-up PR.
Which issue(s) this PR fixes or relates to
Fixes #3806
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]