Update MimirSchedulerQueriesStuck alert and runbook to reflect querier auto-scaling

[treid314]

What this PR does

With our current HPA querier auto-scaling setup it takes sometimes more than a minute for us to detect that we need to scale up and request the additional querier replicas be created. This change allows more time for that to happen before the alert triggers.

Checklist

• [ -] Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

With our current HPA querier auto-scaling setup it takes sometimes more than a minute for us to detect that we need to scale up and request the additional querier replicas be created. This change allows more time for that to happen before the alert triggers.

The alert has a for: 5m, so it needs to fire for 5 consecutive minutes before alerting. The [1m] is just the window to look. Actually increasing the [1m] window potentially increase the chances the alert fires. I don't think this change does what you want.

[treid314]

Even if it's the minimum over that timeframe, not the max? My thought was that timeframe would catch after the queriers have been added and the queues hopefully drop back to 0.

[pracucci]

Even if it's the minimum over that timeframe, not the max? My thought was that timeframe would catch after the queriers have been added and the queues hopefully drop back to 0.

Right, it's the min. Yeah doesn't change anyway. I think you want to increase the for duration.

[pracucci]

Thanks!