Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Commit

Permalink
.maintain/monitoring: Add alert when continuous task ends (#7250)
Browse files Browse the repository at this point in the history
* .maintain/monitoring: Add alert when continuous task ends

Through the `polkadot_tasks_ended_total` Prometheus metric one can tell
when a task ended. Use this metric to alert when specific
known-to-be-continuous tasks end on a node.

* .maintain/monitoring: Don't hard-code task names
  • Loading branch information
mxinden authored Oct 5, 2020
1 parent 6bcc385 commit 3e5ac2a
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions .maintain/monitoring/alerting-rules/alerting-rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,16 @@ groups:
# Others
##############################################################################

- alert: ContinuousTaskEnded
expr: '(polkadot_tasks_spawned_total == 1) - on(instance, task_name)
(polkadot_tasks_ended_total == 1)'
for: 5m
labels:
severity: warning
annotations:
message: 'Continuous task {{ $labels.task_name }} on node
{{ $labels.instance }} ended unexpectedly.'

- alert: AuthorityDiscoveryDiscoveryFailureHigh
expr: 'polkadot_authority_discovery_handle_value_found_event_failure /
ignoring(name)
Expand Down

0 comments on commit 3e5ac2a

Please sign in to comment.