more options for alerts output

[schmurfy]

it would be nice to be able to store them in influxdb to keep an history and it would also be nice to be able to retrieve at a given time a list of active alerts to display them in a frontend or somewhere else.

I am not sure how they are currently stored but it seems rather volatile to me, I want to be able to say in two months: this alert on this hosts was triggered 4 times and here is when and at which value.

Here is an idea of what it might looks taken from the mailing list

var alert = .alert()
                        ... // define levels etc
alert
   .influxDBOut() // send alert data to InfluxDB
      .database('alerts') ....
alert
   .httpOut('alert_events') // create local cache of alert data

[miili]

An implementation of alarm().influxDBOut() eventhandler could look like this:

stream...
    .alert()
        .influxDBOut()
             .database('events')
             .retentionpolicy('default')
             .measurement('alarms') // alarm series
             # Alarm published as values/tags in dedicated alarm series
             .id()      // value from AlertNode
             .tags()    // tags from StreamNode
             .level()   // tag from AlertNode (info/warning/critical)
             .message() // value from AlertNode
             .time()

[barryheap]

+1

[schmurfy]

I started this and still want it but I don't know enough about kapacitor internals or latest influxdb (I use it daily but an old version) to really help move it forward, if someone has a clear idea on how this would look especially on the influxdb side since @miili already gave a first idea on the kapacitor side it would be great.

Here is some ideas of how I would like to query the stored alerts on influxdb:

• how many alerts got triggered but this particular application on a given period
• what are the alerts currently active
• what is the mean/median/min/max resolution time for a specific alert type

[nathanielc]

@schmurfy

#465 now allows for storage of alert data in InfluxDB

how many alerts got triggered by this particular application on a given period

Something like this will now work: select count(level) from alerts where level != 'OK' and time > now() - 5m group by app

what are the alerts currently active

Similarly select last(level) from alerts where group by app

what is the mean/median/min/max resolution time for a specific alert type

This one is a bit harder since calculating the amount of time between points is not possible. But is a requested feature #384

[schmurfy]

great, thanks !