[LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack #940

ronensc · 2021-10-21T15:40:40Z

https://issues.redhat.com/browse/LOG-1815

eranra · 2021-10-24T13:40:41Z

enhancements/cluster-logging/loki-observability.md

+the Loki Deployment. The alerts are based on metrics consumed from Prometheus into alert manager, evaluated and when 
+fired, pushed to notify human operators.
+
+Loki Alerts can be categorized as follows:


@ronensc Can you move into tables?

https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/organizing-information-with-tables

eranra · 2021-10-24T13:41:14Z

enhancements/cluster-logging/loki-observability.md

+
+See also [Defining recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)
+
+##### LokiRequestErrors


same request ... if we can move to tables it will be easier to read

(I propose a table for each alert, with known organized fields)

eranra · 2021-10-24T13:46:58Z

enhancements/cluster-logging/loki-observability.md

+fired, pushed to notify human operators.
+
+Loki Alerts can be categorized as follows:
+* **Loki System**: One or more of the Loki components is misbehaving; this can be due to any of Loki or OCP system that is not working as expected


@ronensc Any chance that we can label those with the relevant components that reported panic ... in the end-of-the-day each Liki component is separate container so it might be good to have in the alert the name of the component that broke?

I installed loki-operator and checked the labels of loki_panic_total metric. It seems that we can distinguish between loki components using the job label

eranra · 2021-10-24T13:48:25Z

enhancements/cluster-logging/loki-observability.md

+* https://monitoring.mixins.dev/loki/
+* https://github.com/rhobs/configuration/blob/main/docs/sop/observatorium.md#lokirequestpanics
+
+##### LokiRequestLatency


Is that for all tenants or maybe affecting only one or more tenants ?

All tenants I guess. I couldn't find a label that distinguishes between labels

eranra · 2021-10-24T13:49:43Z

enhancements/cluster-logging/loki-observability.md

+
+References:
+* https://github.com/rhobs/configuration/blob/main/docs/sop/observatorium.md#lokitenantratelimitwarning
+


There are a lot of missing alerts here,,,, storage, etc ... can we create a tracing table from the name of the alert to the category and severity ... this will help us to see that we have minimal coverage

I added a summary table at the beginning and added a few more alerts

eranra · 2021-10-25T11:17:19Z

enhancements/cluster-logging/loki-observability.md

+
+##### LokiRequestLatency
+
+The 99th percentile is experiencing high latency.


Please extend to explain also bigger than 1

eranra · 2021-10-25T11:18:04Z

enhancements/cluster-logging/loki-observability.md

+namespace_job_route:loki_request_duration_seconds:99quantile{route!~"(?i).*tail.*"} > 1
+```
+
+underline recording rule:


Please explain what is the purpose of the metric created by this recording rule

eranra · 2021-10-25T11:18:34Z

enhancements/cluster-logging/loki-observability.md

+
+##### LokiTenantRateLimitWarning
+
+A tenant is experiencing rate limiting.


Please enhance to explain the limit (>100)

eranra · 2021-10-27T08:33:17Z

@ronensc and @periklis I found this page https://pages.github.ibm.com/CloudEngineering/system_architecture/guardrails_standards/monitoring_maturity.md that provides generic but very useful information on needed alerts and dashboards for services such as Loki ... this is used by all IBM service teams so I think we can learn from that a lot ... @ronensc please add those to the list ... I am not saying that we need to implement all of those for day zero .. but we can add them and discuss what we implement and what not (and when).

alanconway

Suggested some minor name changes, otherwise LGTM.

enhancements/cluster-logging/loki-observability.md

ronensc · 2021-11-23T14:43:23Z

@alanconway @periklis @eranra
Since Loki is just a logging system, is there an alert that is justified a critical severity? (i.e. warrants waking someone in the middle of the night)

alanconway

Need to squash commits before merge, otherwise LGTM.

periklis

/lgtm

openshift-ci · 2021-12-06T10:06:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alanconway, periklis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~enhancements/cluster-logging/OWNERS~~ [alanconway,periklis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2021

openshift-ci bot requested review from alanconway and jcantrill October 21, 2021 15:50

eranra reviewed Oct 24, 2021

View reviewed changes

eranra reviewed Oct 25, 2021

View reviewed changes

alanconway suggested changes Nov 18, 2021

View reviewed changes

enhancements/cluster-logging/loki-observability.md Outdated Show resolved Hide resolved

enhancements/cluster-logging/loki-observability.md Outdated Show resolved Hide resolved

ronensc changed the title ~~[WIP] [LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack~~ [LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack Nov 24, 2021

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 24, 2021

alanconway approved these changes Nov 26, 2021

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 26, 2021

Add enhancment proposal: LokiStack Observability

ca7bed0

ronensc force-pushed the loki-observability branch from 083547e to ca7bed0 Compare December 1, 2021 13:32

periklis approved these changes Dec 6, 2021

View reviewed changes

openshift-ci bot assigned periklis Dec 6, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 6, 2021

openshift-merge-robot merged commit afea709 into openshift:master Dec 6, 2021

ronensc deleted the loki-observability branch January 9, 2022 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack #940

[LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack #940

ronensc commented Oct 21, 2021

eranra Oct 24, 2021 •

edited

Loading

ronensc Nov 2, 2021

eranra Oct 24, 2021 •

edited

Loading

ronensc Nov 2, 2021

eranra Oct 24, 2021

ronensc Nov 2, 2021

eranra Oct 24, 2021

ronensc Nov 2, 2021

eranra Oct 24, 2021

ronensc Nov 2, 2021

eranra Oct 25, 2021

ronensc Nov 2, 2021

eranra Oct 25, 2021

ronensc Nov 2, 2021

eranra Oct 25, 2021

eranra commented Oct 27, 2021

alanconway left a comment

ronensc commented Nov 23, 2021

alanconway left a comment

periklis left a comment

openshift-ci bot commented Dec 6, 2021


		See also [Defining recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/)

		##### LokiRequestErrors


		References:
		* https://github.com/rhobs/configuration/blob/main/docs/sop/observatorium.md#lokitenantratelimitwarning


		##### LokiRequestLatency

		The 99th percentile is experiencing high latency.


		##### LokiTenantRateLimitWarning

		A tenant is experiencing rate limiting.

[LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack #940

[LOG-1815] Enhancement proposal: Add alerts and rules for operator-managed LokiStack #940

Conversation

ronensc commented Oct 21, 2021

eranra Oct 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eranra Oct 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eranra commented Oct 27, 2021

alanconway left a comment

Choose a reason for hiding this comment

ronensc commented Nov 23, 2021

alanconway left a comment

Choose a reason for hiding this comment

periklis left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Dec 6, 2021

eranra Oct 24, 2021 •

edited

Loading

eranra Oct 24, 2021 •

edited

Loading