Support for DNS discovery for alertmanagers to find peers in a cluster

[samitpal]

It might be a good addition to add DNS discovery capability in multi-tenant alertmanager to discover peers. Right now there is a -cluster.peer flag which can be used multiple times but that has to be done statically. I tried with a dns host with multiple IP targets with -cluster.peer but it seems like the dns name does not get re-resolved when a host goes down.

[pracucci]

Could you check if #2851 solve your issue, please?

I tried with a dns host with multiple IP targets with -cluster.peer but it seems like the dns name does not get re-resolved when a host goes down.

@gotjosh Are you aware of this?

[gotjosh]

I think this is slightly different but can use similar mechanisms. I've added the capability to discover Alertmanagers from the ruler, this is about discovering other Alertmanagers from the Alertmanager itself.

[gotjosh]

Could you please assign this to me @pracucci?

[pracucci]

Could you please assign this to me @pracucci?

Definitely, done! 🙏

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[gotjosh]

Do we still want this? Moving forward, I'm not sure I see a point in improving the old clustering form.

[pracucci]

Do we still want this? Moving forward, I'm not sure I see a point in improving the old clustering form.

I think the way ahead is the ring-based alertmanager scalability, which is what we're currently working on.

[jtlisi]

Closing this because we will be solving the use case of this issue using a ring-based scalable alertmanager