[Proposal] Scalable Alertmanager #3574

gotjosh · 2020-12-08T17:39:41Z

What this PR does:

Introduces a series of changes for scaling the Alertmanager. Given the proposal is quite lengthy, I have tried to keep it to the point on most parts.

Please do let me know if there's an area that requires expanding.

Signed-off-by: gotjosh josue@grafana.com

Checklist

N/A Tests updated
Documentation added
N/A CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Introduces a series of changes for scaling the Alertmanager. Given the proposal is quite lengthy, I have tried to keep it to the point on most parts. Please do let me know if there's any area that requires expanding. Signed-off-by: gotjosh <josue@grafana.com>

Signed-off-by: gotjosh <josue@grafana.com>

gotjosh · 2020-12-08T17:46:29Z

docs/proposals/scalable-alertmanager.md

+We can either run it as a separate service or embed it. **I propose we simply embed it**. At its core it’ll be simpler to operate. With future work making it possible to run as a separate service so that operators can scale when/if needed.
+
+
+## Conclusion


One thing I have not talked about is sequencing - is it worth mentioning that now? My plan is to leave the current alertmanager struct untouched and create a new one can be specified via a flag. The PRs can be slices of the sections described. Keeping in mind that I can break those down as I deemed fit to prioritise simpler and less risky reviews.

I'd be interested in your ideas for sequencing, but seems like it could be separate from this PR.

I'm prepping the first PR, I'll tag you in it so that you can get an idea of what my plan is.

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci

We discussed offline and LGTM, thanks! I just pushed a commit to fix the image location.

jtlisi · 2020-12-09T17:11:06Z

docs/proposals/scalable-alertmanager.md

+
+![Scalable Alertmanager Architecture](/images/proposals/scalable-am.png)
+
+**POST /api/v1/alerts (from the ruler) can go to any Alertmanager replica.** The AM distributor uses the ring to write alerts to a quorum of AM managers (reusing the existing code). We continue to use the same in-memory data structure from the upstream Alertmanager to save alerts and notify other pieces


question: In this architecture the ruler will still use HTTP to forward alerts to the Alertmanager. In order to avoid multiple deliveries will the Ruler only be configured with a single address? If so does this assume the user will need to place a loadbalancer between the ruler and the am-distributor to achieve an equal distribution of traffic and ensure availability?

I think the short answer is yes.

Just like today, you need to use a headless service (or include all the alertmanager URLs manually) to avoid load balancing traffic between Alertmanagers. You'll need to ensure you have a way to load balance traffic between the ruler <> Alertmanager.

jtlisi

LGTM

ranton256

The proposal LGTM. Thanks for doing this.

ranton256 · 2020-12-17T15:49:04Z

docs/proposals/scalable-alertmanager.md

+
+To achieve horizontal scalability, we need to distribute the workload among replicas of the service. We need to choose an appropriate field to use to distribute the workload. The field must be present on all the API requests to the Alertmanager service.
+
+**We propose the sharding on Tenant ID**. The simplicity of this implementation, would allow us to get up and running relatively quickly whilst helping us validate assumptions. We intend to use the existing ring code to manage this. Other options such as tenant ID + receiver or Tenant ID + route are relatively complex as distributor components (in this case the Ruler) would need to be aware of Alertmanager configuration.


I think this con is fine for the time being If the current design can support ~2000 average tenants, it could with a little handholding support a tenant 2000 times normal on by fiddling with sharding overrides, so I don't think this limitation is reason enough to not choose the simple option 1, shard on tenant id.

ranton256 · 2020-12-17T15:50:46Z

docs/proposals/scalable-alertmanager.md

+We can either run it as a separate service or embed it. **I propose we simply embed it**. At its core it’ll be simpler to operate. With future work making it possible to run as a separate service so that operators can scale when/if needed.
+
+
+## Conclusion


I'd be interested in your ideas for sequencing, but seems like it could be separate from this PR.

pracucci · 2020-12-17T16:42:33Z

Going to merge it. As usual, we'll address any post-merge comment.

The first part of the proposed as part of cortexproject#3574, introduces sharding via the ring for the Alertmanager component. Signed-off-by: gotjosh <josue@grafana.com>

* Alertmanager: Allow sharding of alertmanager tenants The first part of the proposed as part of #3574, introduces sharding via the ring for the Alertmanager component. Signed-off-by: gotjosh <josue@grafana.com> * Appease the linter Signed-off-by: gotjosh <josue@grafana.com> * Update CHANGELOG to warn about Alertmanager sharding Signed-off-by: gotjosh <josue@grafana.com> * Fix, last typo. Signed-off-by: gotjosh <josue@grafana.com>

* Alertmanager: Allow sharding of alertmanager tenants The first part of the proposed as part of cortexproject/cortex#3574, introduces sharding via the ring for the Alertmanager component. Signed-off-by: gotjosh <josue@grafana.com> * Appease the linter Signed-off-by: gotjosh <josue@grafana.com> * Update CHANGELOG to warn about Alertmanager sharding Signed-off-by: gotjosh <josue@grafana.com> * Fix, last typo. Signed-off-by: gotjosh <josue@grafana.com> Former-commit-id: 3aba107

* Alertmanager: Allow sharding of alertmanager tenants The first part of the proposed as part of cortexproject/cortex#3574, introduces sharding via the ring for the Alertmanager component. Signed-off-by: gotjosh <josue@grafana.com> * Appease the linter Signed-off-by: gotjosh <josue@grafana.com> * Update CHANGELOG to warn about Alertmanager sharding Signed-off-by: gotjosh <josue@grafana.com> * Fix, last typo. Signed-off-by: gotjosh <josue@grafana.com>

gotjosh added 2 commits December 8, 2020 17:38

Fix image link

1119091

Signed-off-by: gotjosh <josue@grafana.com>

gotjosh commented Dec 8, 2020

View reviewed changes

Fixed image and white noise

6a4799c

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci approved these changes Dec 9, 2020

View reviewed changes

jtlisi reviewed Dec 9, 2020

View reviewed changes

jtlisi approved these changes Dec 9, 2020

View reviewed changes

ranton256 approved these changes Dec 17, 2020

View reviewed changes

pracucci merged commit 48e4ae3 into cortexproject:master Dec 17, 2020

gotjosh mentioned this pull request Jan 8, 2021

Alertmanager: Allow sharding of alertmanager tenants #3664

Merged

8 tasks

gotjosh mentioned this pull request Feb 18, 2021

Alertmanager: Replicate state using the Ring #3839

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Proposal] Scalable Alertmanager #3574

[Proposal] Scalable Alertmanager #3574

gotjosh commented Dec 8, 2020 •

edited

Loading

gotjosh Dec 8, 2020

ranton256 Dec 17, 2020

gotjosh Dec 17, 2020

pracucci left a comment

jtlisi Dec 9, 2020

gotjosh Dec 9, 2020

jtlisi Dec 9, 2020

jtlisi left a comment

ranton256 left a comment

ranton256 Dec 17, 2020

ranton256 Dec 17, 2020

pracucci commented Dec 17, 2020

		We can either run it as a separate service or embed it. I propose we simply embed it. At its core it’ll be simpler to operate. With future work making it possible to run as a separate service so that operators can scale when/if needed.


		## Conclusion


		![Scalable Alertmanager Architecture](/images/proposals/scalable-am.png)

		POST /api/v1/alerts (from the ruler) can go to any Alertmanager replica. The AM distributor uses the ring to write alerts to a quorum of AM managers (reusing the existing code). We continue to use the same in-memory data structure from the upstream Alertmanager to save alerts and notify other pieces


		To achieve horizontal scalability, we need to distribute the workload among replicas of the service. We need to choose an appropriate field to use to distribute the workload. The field must be present on all the API requests to the Alertmanager service.

		We propose the sharding on Tenant ID. The simplicity of this implementation, would allow us to get up and running relatively quickly whilst helping us validate assumptions. We intend to use the existing ring code to manage this. Other options such as tenant ID + receiver or Tenant ID + route are relatively complex as distributor components (in this case the Ruler) would need to be aware of Alertmanager configuration.

[Proposal] Scalable Alertmanager #3574

[Proposal] Scalable Alertmanager #3574

Conversation

gotjosh commented Dec 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtlisi left a comment

Choose a reason for hiding this comment

ranton256 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci commented Dec 17, 2020

gotjosh commented Dec 8, 2020 •

edited

Loading