Merge pull request #335 from grafana/playbooks-for-config-alerts

Fixed and improved runtime config alerts and playbooks

CHANGELOG.md

@@ -14,9 +14,12 @@
 * [CHANGE] Ingester/Ruler: set `-server.grpc-max-send-msg-size-bytes` and `-server.grpc-max-send-msg-size-bytes` to sensible default values (10MB). #326
 * [CHANGE] Renamed `CortexCompactorHasNotUploadedBlocksSinceStart` to `CortexCompactorHasNotUploadedBlocks`. #334
 * [CHANGE] Renamed `CortexCompactorRunFailed` to `CortexCompactorHasNotSuccessfullyRunCompaction`. #334
+* [CHANGE] Renamed `CortexInconsistentConfig` alert to `CortexInconsistentRuntimeConfig` and increased severity to `critical`. #335
+* [CHANGE] Increased `CortexBadRuntimeConfig` alert severity to `critical` and removed support for `cortex_overrides_last_reload_successful` metric (was removed in Cortex 1.3.0). #335
 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317
 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
+* [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335
 
 ## 1.9.0 / 2021-05-18
 

cortex-mixin/alerts/alerts.libsonnet

@@ -92,39 +92,30 @@
           },
         },
         {
-          alert: 'CortexInconsistentConfig',
+          alert: 'CortexInconsistentRuntimeConfig',
           expr: |||
-            count(count by(%s, job, sha256) (cortex_config_hash)) without(sha256) > 1
+            count(count by(%s, job, sha256) (cortex_runtime_config_hash)) without(sha256) > 1
           ||| % $._config.alert_aggregation_labels,
           'for': '1h',
           labels: {
-            severity: 'warning',
+            severity: 'critical',
           },
           annotations: {
             message: |||
-              An inconsistent config file hash is used across cluster {{ $labels.job }}.
+              An inconsistent runtime config file is used across cluster {{ $labels.job }}.
             |||,
           },
         },
         {
-          // As of https://github.com/cortexproject/cortex/pull/2092, this metric is
-          // only exposed when it is supposed to be non-zero, so we don't need to do
-          // any special filtering on the job label.
-          // The metric itself was renamed in
-          // https://github.com/cortexproject/cortex/pull/2874
-          //
-          // TODO: Remove deprecated metric name of
-          // cortex_overrides_last_reload_successful in the future
           alert: 'CortexBadRuntimeConfig',
           expr: |||
+            # The metric value is reset to 0 on error while reloading the config at runtime.
             cortex_runtime_config_last_reload_successful == 0
-              or
-            cortex_overrides_last_reload_successful == 0
           |||,
           // Alert quicker for human errors.
           'for': '5m',
           labels: {
-            severity: 'warning',
+            severity: 'critical',
           },
           annotations: {
             message: |||

cortex-mixin/docs/playbooks.md

@@ -367,13 +367,33 @@ _TODO: this playbook has not been written yet._
 
 _TODO: this playbook has not been written yet._
 
-### CortexInconsistentConfig
+### CortexInconsistentRuntimeConfig
 
-_TODO: this playbook has not been written yet._
+This alert fires if multiple replicas of the same Cortex service are using a different runtime config for a longer period of time.
+
+The Cortex runtime config is a config file which gets live reloaded by Cortex at runtime. In order for Cortex to work properly, the loaded config is expected to be the exact same across multiple replicas of the same Cortex service (eg. distributors, ingesters, ...). When the config changes, there may be short periods of time during which some replicas have loaded the new config and others are still running on the previous one, but it shouldn't last for more than few minutes.
+
+How to **investigate**:
+- Check how many different config file versions (hashes) are reported
+  ```
+  count by (sha256) (cortex_runtime_config_hash{namespace="<namespace>"})
+  ```
+- Check which replicas are running a different version
+  ```
+  cortex_runtime_config_hash{namespace="<namespace>",sha256="<unexpected>"}
+  ```
+- Check if the runtime config has been updated on the affected replicas' filesystem. Check `-runtime-config.file` command line argument to find the location of the file.
+- Check the affected replicas logs and look for any error loading the runtime config
 
 ### CortexBadRuntimeConfig
 
-_TODO: this playbook has not been written yet._
+This alert fires if Cortex is unable to reload the runtime config.
+
+This typically means an invalid runtime config was deployed. Cortex keeps running with the previous (valid) version of the runtime config; running Cortex replicas and the system availability shouldn't be affected, but new replicas won't be able to startup until the runtime config is fixed.
+
+How to **investigate**:
+- Check the latest runtime config update (it's likely to be broken)
+- Check Cortex logs to get more details about what's wrong with the config
 
 ### CortexQuerierCapacityFull