Better handling of event consumer lag alerts in Datadog

[robrap]

The consumer lag alerts from Datadog are all going to arch-bom. They have not been split by team and consumer, with safety-net alerts, as was done for the New Relic side alerts.

Acceptance Criteria:

• A new example monitor of a consumer lag issue that is filtered by the consumer_group_id
  • Docs currently discuss splitting by topic, but we should probably split by consumer_group_id
• A catch-all monitor that will alert us when any particular topic falls behind for 2 hours
  • We can update the existing monitor to behave this way.
• Update docs with new approach

Note:

• Docs on squad-based Datadog alerts: https://2u-internal.atlassian.net/wiki/spaces/AT/pages/585400611/How+to+set+up+squad-based+event+bus+alerting#Datadog-Alerts
• See example alert details: https://2u-internal.app.opsgenie.com/alert/detail/39f17272-d5e4-45cc-87e7-97a7c7db5bbc-1706023992011/details
  • Datadog Tags: portfolio:edx, group:arch-bom, confluent_cluster:edx-stage, consumer_group_id:credentials_stage, monitor, topic:stage-learning-certificate-lifecycle
• See New Relic alerts for event bus for examples.

[dianakhuang]

New alert split on consumer group id for edxapp: https://app.datadoghq.com/monitors/139126405?view=spans

[dianakhuang]

We believe that the existing monitor does work as a catch all currently. The reason why it didn't trigger on a particular error is that Confluent wasn't sending us the proper data to trigger the alert.

[dianakhuang]

• Create a new opsgenie integration to alert Phoenix on the course discovery lag monitor
• Update documentation to have examples of both topic-based and consumer group alerting (edx-platform vs. separate service)
• Update runbooks so that they match the actual alerts.
• Add a warning box that maybe this will change as we understand Datadog better.