Add a query parameter to test notification channels

[mariliamelo]

TL;DR

Consider adding a query param to the health endpoint that fires an alert and enables testing the notification channels.

Additional information

We can create custom metrics from logging and use it in monitoring
you can alert on "last value". The problem you have to be careful about is making sure that this metric always has a value. If it goes from over threshold to no data, the alert won't recover until the next time you have good data

[sethvargo]

Added the scaffolding for this in #1187 (can be merged as-is). You'll be able to hit /healthz?service=alerts. I'm gonna defer the actual alert and its notification setup to yall.

[sethvargo]

@mariliamelo @yegle is this still something we want to do? We've pushed it off for multiple milestones now.

[sethvargo]

Talked with Mike and we are not going to do this. It gives anyone on the Internet the ability to page us. We'll discuss the use case internally and come up with alternatives.

/close

[google-oss-robot]

@sethvargo: Closing this issue.

In response to this:

Talked with Mike and we are not going to do this. It gives anyone on the Internet the ability to page us. We'll discuss the use case internally and come up with alternatives.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.