PagerDuty Service Enhancement

[rossmcdonald]

Based on the PagerDuty event documentation, I believe Kapacitor should be sending trigger events when an issue is opened, and resolve when an issue is marked resolved. From looking at the current code, it appears only trigger alerts are sent. This means that a new trigger event will be opened for every state change in the alert. Also, it doesn't look like the incident_key value is stored or sent when the issue is opened or resolved.

Since PagerDuty events are tertiary (they are only ever trigger, acknowledged, or resolved), I that think that it makes sense to take a binary alerting approach when alerting through PagerDuty. Namely, I think that when an alert is set to either WARN or CRITICAL that a trigger alert is fired, and, when an alert is set to OK, a resolve event should fire (using the same incident_key). Multiple triggers to the same incident_key will only append themselves to the trigger log, so there should be no issue with executing multiple WARN or CRITICAL events.

@nathanielc What do you think? I'd be willing to take this on, as long as you don't have any objections.

[nathanielc]

Go for it, seems like you have a good understanding. Also you might want to take a look at this one too, #82