Merge pull request #404 from grafana/improve-CortexIngesterReachingSe…

…riesLimit Fine tune CortexIngesterReachingSeriesLimit alert
grafana · Oct 14, 2021 · 189e0c7 · 189e0c7
2 parents db05c86 + be7638b
commit 189e0c7
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -32,6 +32,7 @@
   * `_config` > `queryFrontend` > `sharded_queries_enabled`
   * `_config` > `queryFrontend` > `query_split_factor`
 * [CHANGE] Split `cortex_api` recording rule group into three groups. This is a workaround for large clusters where this group can become slow to evaluate. #401
+* [CHANGE] Increased `CortexIngesterReachingSeriesLimit` warning threshold from 70% to 80% and critical threshold from 85% to 90%. #404
 * [ENHANCEMENT] Add overrides config to compactor. This allows setting retention configs per user. #386
 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317
 * [ENHANCEMENT] Cortex-mixin: Include `cortex-gw-internal` naming variation in default `gateway` job names. #328

diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet
@@ -260,7 +260,7 @@
                 (cortex_ingester_memory_series / ignoring(limit) cortex_ingester_instance_limits{limit="max_series"})
                 and ignoring (limit)
                 (cortex_ingester_instance_limits{limit="max_series"} > 0)
-            ) > 0.7
+            ) > 0.8
           |||,
           'for': '3h',
           labels: {
@@ -279,7 +279,7 @@
                 (cortex_ingester_memory_series / ignoring(limit) cortex_ingester_instance_limits{limit="max_series"})
                 and ignoring (limit)
                 (cortex_ingester_instance_limits{limit="max_series"} > 0)
-            ) > 0.85
+            ) > 0.9
           |||,
           'for': '5m',
           labels: {