-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[R&D] Data flow stopped rule #141936
Comments
Pinging @elastic/actionable-observability (Team: Actionable Observability) |
Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI) |
We've been talking about NO_DATA indications since when alerting started, but never came up with how we would actually do this. I think the rule executors would need to return an indication of this, since the framework can't figure it out. Should it be a new action group, like I'm not sure how long it would take us to do this; also not sure how it would interact with the current "NO DATA" action group already used by at least metric threshold. I'd think we'd want to make sure we could migrate from that existing one, to the new one, or just "make it work" with the existing one (even better). |
As a follow up to the investigation done here, we've identified a need to be able to define a rule that checks if data stops coming in from an expected source.
In the Stack Monitoring case we have regular SDHs where data stops coming in for some reason (config changes, cluster upgrades, outages or cluster overload), it would be great if we had a flexible way to create rules around such situations.
I'm not certain about what kind of granularity we should look at (pure document rate into an index/data stream with awareness of sending frequency, splitting by metricset, etc.).
Ideally, this rule would be made in a way that it can be used no matter what the source of data is, as long as we expect it to be coming into Elasticsearch we should alert if it stops (with some config for how long an accepted delay might be).
The text was updated successfully, but these errors were encountered: