Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services: add maintenance mode option to snooze alert notifications #2512

Merged
merged 43 commits into from
Jul 27, 2022

Conversation

Forfold
Copy link
Contributor

@Forfold Forfold commented Jul 18, 2022

Closes #2490

Adds a button to configure maintenance mode for 1, 2, or 4 hours from any service details page. All notifications during this time period will be silenced, and escalations will not be processed. Any incoming alerts will still be created (snoozing integration keys out of scope).

After maintenance mode ends, alerts will be sent out as normal to the on-call user.

Screenshots:

Service details maintenance mode button
Maintenance mode dialog
Maintenance mode notice banners


Escalate alert GraphQL error while in maintenance mode

Forfold and others added 30 commits July 5, 2022 12:54
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: Nathaniel Cook <NathanielJCook@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: Nathaniel Cook <NathanielJCook@outlook.com>
Co-authored-by: Nathaniel Cook <NathanielJCook@outlook.com>
… disable escalating if maintenance mode exists

Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: mastercactapus <mastercactapus@gmail.com>
…on cancel

Co-authored-by: tony-tvu <Tony.T.Vu@outlook.com>
Co-authored-by: mastercactapus <mastercactapus@gmail.com>
Copy link
Member

@mastercactapus mastercactapus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the escalation code, I had a couple of thoughts, what's the expected behavior when escalating multiple alerts when only one or two are in maintenance mode?

When we talked about it, we figured an error should be returned, but the method signatures suggest we should instead have it do nothing, as it's supposed to return only the alerts that were escalated.

escalateAlerts(input: [1]) -> []

As it stands, if one alert is for a service in maintenance mode, all will fail with a generic error. It's more a limitation of how the API is set up currently.

I'll reach out to chat more about it

@Forfold
Copy link
Contributor Author

Forfold commented Jul 18, 2022

@mastercactapus

For the escalation code, I had a couple of thoughts, what's the expected behavior when escalating multiple alerts when only one or two are in maintenance mode?

totally did not think of this use case, we were thinking in terms of escalateAlerts(input: [id]) the whole time since we were only testing with one alert. I think it makes sense to return maybe an array of true/error or something similar?

alert 1: active service
alert 2: maint. mode service

escalateAlerts(ids: [1, 2])

returns

{
  "errors": [{
    message: "escalate alert 2: in maintenance mode",
    path: ['escalateAlerts']
  }],
  "data": {
    "1": true
  }
}

Forfold and others added 7 commits July 19, 2022 08:26
Co-authored-by: Nathaniel Caza <mastercactapus@gmail.com>
Co-authored-by: Nathaniel Caza <mastercactapus@gmail.com>
Co-authored-by: Nathaniel Caza <mastercactapus@gmail.com>
Co-authored-by: Nathaniel Caza <mastercactapus@gmail.com>
Co-authored-by: Nathaniel Caza <mastercactapus@gmail.com>
@Forfold Forfold requested a review from mastercactapus July 20, 2022 17:02
mastercactapus
mastercactapus previously approved these changes Jul 25, 2022
mastercactapus
mastercactapus previously approved these changes Jul 25, 2022
@mastercactapus mastercactapus merged commit 31cdeb0 into master Jul 27, 2022
@mastercactapus mastercactapus deleted the service-maintenance branch July 27, 2022 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

service maintenance: pause outgoing notifications
4 participants