Alert Message send failed

[seaurching]

Describe the bug

The alert message send filed while rules number above 1. And it show

level=error ts=2022-08-28T05:31:00.097461Z caller=dispatch.go:354 component=MultiTenantAlertmanager user=anonymous component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="invalid service state: Terminated, expected: Running"

Image

Version

Mimir, version 2.2.0 (branch: HEAD, revision: 65344e2ed)
  go version:       go1.17.8
  platform:         darwin/amd64

[dimitarvdimitrov]

the log message you posted - do you see this in grafana, in the ruler logs, or in the alertmanager logs?

[chriskuchin]

I have a similar error and I see mine within the mimir alertmanager logs

The alertmanager runs fine for a few minutes but then that log starts and just spams my logs and further alerts fail to send

[dimitarvdimitrov]

I can see this happening in our alertmanager. But this is usually preceeded by either deleting the alerting configuration for a tenant or resharding - adding or removing AM replicas or an AM replica restarting. Most importantly, it should only last a few seconds.

In your cases how long does this "spamming" last - a few seconds, a minute, an hour? Until restart?

Also, what are the Alertmanager logs preceding this "spamming?"

[chriskuchin]

So I made a few adjustments to my config increasing replicas of the alertmanager and increasing the log level it was originally warn which mean preceding the problem there were no logs.

It hasn't repeated since then

[dimitarvdimitrov]

what replication factor were you running with when it was happening @chriskuchin ? I mean alertmanager.sharding_ring.replication_factor

[chriskuchin]

Was running with a factor of 1 and adjusted to 2

…

On Fri, Sep 9, 2022, at 1:44 AM, Dimitar Dimitrov wrote: what replication factor were you running with when it was happening @chriskuchin <https://github.com/chriskuchin> ? I mean `alertmanager.sharding_ring.replication_factor` — Reply to this email directly, view it on GitHub <#2873 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANNV5F3CQFLRVKTHQTV2V4DV5LTE5ANCNFSM6AAAAAAQBSRHNE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dimitarvdimitrov]

thanks. looks like it's an issue with replication_factor == 1 we are tracking this in #2910

[chriskuchin]

Awesome thanks good to know

…

On Fri, Sep 9, 2022, at 9:03 AM, Dimitar Dimitrov wrote: thanks. looks like it's an issue with `replication_fgactor == 1` we are tracking this in #2910 <#2910> — Reply to this email directly, view it on GitHub <#2873 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANNV5FZCTMYFLT2GFPWVSCTV5NGUFANCNFSM6AAAAAAQBSRHNE>. You are receiving this because you were mentioned.Message ID: ***@***.***>