Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Span the annotation.message in alerts as YAML multiline strings. #412

Merged
merged 5 commits into from
Nov 2, 2021
Merged

Span the annotation.message in alerts as YAML multiline strings. #412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
33dd2e9
multiline annotation for CortexIngesterUnhealthy alert
aallawala Oct 18, 2021
929623f
add changelog entry
aallawala Oct 18, 2021
181f725
use multiline wrapper for the rest of annotation.message templates
aallawala Oct 18, 2021
8ad55b9
update changelog
aallawala Oct 18, 2021
d6bdd88
add in compactor|blocks.libsonnet as well
aallawala Oct 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
* [BUGFIX] Upstream recording rule `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` renamed. #379
* [BUGFIX] Treat `compactor_blocks_retention_period` type as string rather than int.#395
* [BUGFIX] Fixed writes/reads/alertmanager resources dashboards to use `$._config.job_names.gateway`. #403
* [BUGFIX] Span the annotation.message in alerts as YAML multiline strings. #412

## 1.9.0 / 2021-05-18

Expand Down
20 changes: 15 additions & 5 deletions cortex-mixin/alerts/alerts.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex cluster %(alert_aggregation_variables)s has {{ printf "%%f" $value }} unhealthy ingester(s).' % $._config,
message: |||
Cortex cluster %(alert_aggregation_variables)s has {{ printf "%%f" $value }} unhealthy ingester(s).
||| % $._config,
},
},
{
Expand Down Expand Up @@ -199,7 +201,9 @@
severity: 'warning',
},
annotations: {
message: '{{ $labels.job }}/{{ $labels.instance }} has restarted {{ printf "%.2f" $value }} times in the last 30 mins.',
message: |||
{{ $labels.job }}/{{ $labels.instance }} has restarted {{ printf "%.2f" $value }} times in the last 30 mins.
|||,
},
},
{
Expand All @@ -212,7 +216,9 @@
severity: 'critical',
},
annotations: {
message: '{{ $labels.job }}/{{ $labels.instance }} transfer failed.',
message: |||
{{ $labels.job }}/{{ $labels.instance }} transfer failed.
|||,
},
},
{
Expand Down Expand Up @@ -266,7 +272,9 @@
severity: 'critical',
},
annotations: {
message: '{{ $labels.job }}/{{ $labels.instance }} has a number of mmap-ed areas close to the limit.',
message: |||
{{ $labels.job }}/{{ $labels.instance }} has a number of mmap-ed areas close to the limit.
|||,
},
},
],
Expand Down Expand Up @@ -688,7 +696,9 @@
severity: 'warning',
},
annotations: {
message: 'Cortex instance {{ $labels.instance }} in %(alert_aggregation_variables)s sees incorrect number of gossip members.' % $._config,
message: |||
Cortex instance {{ $labels.instance }} in %(alert_aggregation_variables)s sees incorrect number of gossip members.
||| % $._config,
},
},
],
Expand Down
60 changes: 45 additions & 15 deletions cortex-mixin/alerts/blocks.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has not shipped any block in the last 4 hours.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has not shipped any block in the last 4 hours.
||| % $._config,
},
},
{
Expand All @@ -43,7 +45,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has not shipped any block in the last 4 hours.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has not shipped any block in the last 4 hours.
||| % $._config,
},
},
{
Expand All @@ -61,7 +65,9 @@
severity: 'critical',
},
annotations: {
message: "Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has compacted a block {{ $value | humanizeDuration }} ago but it hasn't been successfully uploaded to the storage yet." % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s has compacted a block {{ $value | humanizeDuration }} ago but it hasn't been successfully uploaded to the storage yet.
||| % $._config,
},
},
{
Expand All @@ -77,7 +83,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to compact TSDB head.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to compact TSDB head.
||| % $._config,
},
},
{
Expand All @@ -89,7 +97,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to truncate TSDB head.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to truncate TSDB head.
||| % $._config,
},
},
{
Expand All @@ -101,7 +111,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to create TSDB checkpoint.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to create TSDB checkpoint.
||| % $._config,
},
},
{
Expand All @@ -113,7 +125,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to delete TSDB checkpoint.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to delete TSDB checkpoint.
||| % $._config,
},
},
{
Expand All @@ -125,7 +139,9 @@
severity: 'warning',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to truncate TSDB WAL.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to truncate TSDB WAL.
||| % $._config,
},
},
{
Expand All @@ -137,7 +153,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s got a corrupted TSDB WAL.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s got a corrupted TSDB WAL.
||| % $._config,
},
},
{
Expand All @@ -150,7 +168,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to write to TSDB WAL.' % $._config,
message: |||
Cortex Ingester {{ $labels.instance }} in %(alert_aggregation_variables)s is failing to write to TSDB WAL.
||| % $._config,
},
},
{
Expand All @@ -166,7 +186,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Querier {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully scanned the bucket since {{ $value | humanizeDuration }}.' % $._config,
message: |||
Cortex Querier {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully scanned the bucket since {{ $value | humanizeDuration }}.
||| % $._config,
},
},
{
Expand All @@ -190,7 +212,9 @@
severity: 'warning',
},
annotations: {
message: 'Cortex Queries in %(alert_aggregation_variables)s are refetching series from different store-gateways (because of missing blocks) for the {{ printf "%%.0f" $value }}%% of queries.' % $._config,
message: |||
Cortex Queries in %(alert_aggregation_variables)s are refetching series from different store-gateways (because of missing blocks) for the {{ printf "%%.0f" $value }}%% of queries.
||| % $._config,
},
},
{
Expand All @@ -206,7 +230,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Store Gateway {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully synched the bucket since {{ $value | humanizeDuration }}.' % $._config,
message: |||
Cortex Store Gateway {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully synched the bucket since {{ $value | humanizeDuration }}.
||| % $._config,
},
},
{
Expand All @@ -219,7 +245,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex bucket index for tenant {{ $labels.user }} in %(alert_aggregation_variables)s has not been updated since {{ $value | humanizeDuration }}.' % $._config,
message: |||
Cortex bucket index for tenant {{ $labels.user }} in %(alert_aggregation_variables)s has not been updated since {{ $value | humanizeDuration }}.
||| % $._config,
},
},
{
Expand All @@ -233,7 +261,9 @@
severity: 'warning',
},
annotations: {
message: 'Cortex tenant {{ $labels.user }} in %(alert_aggregation_variables)s has {{ $value }} partial blocks.' % $._config,
message: |||
Cortex tenant {{ $labels.user }} in %(alert_aggregation_variables)s has {{ $value }} partial blocks.
||| % $._config,
},
},
],
Expand Down
24 changes: 18 additions & 6 deletions cortex-mixin/alerts/compactor.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully cleaned up blocks in the last 6 hours.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not successfully cleaned up blocks in the last 6 hours.
||| % $._config,
},
},
{
Expand All @@ -30,7 +32,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not run compaction in the last 24 hours.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not run compaction in the last 24 hours.
||| % $._config,
},
},
{
Expand All @@ -44,7 +48,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not run compaction in the last 24 hours.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not run compaction in the last 24 hours.
||| % $._config,
},
},
{
Expand All @@ -57,7 +63,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s failed to run 2 consecutive compactions.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s failed to run 2 consecutive compactions.
||| % $._config,
},
},
{
Expand All @@ -73,7 +81,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not uploaded any block in the last 24 hours.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not uploaded any block in the last 24 hours.
||| % $._config,
},
},
{
Expand All @@ -87,7 +97,9 @@
severity: 'critical',
},
annotations: {
message: 'Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not uploaded any block in the last 24 hours.' % $._config,
message: |||
Cortex Compactor {{ $labels.instance }} in %(alert_aggregation_variables)s has not uploaded any block in the last 24 hours.
||| % $._config,
},
},
],
Expand Down