[Alerting] Telemetry for potential rule execution guardrails

[mikecote]

Before implementing guardrails and limitations to the alerting rules, we should gather data to validate where guardrails are necessary in relation to the rule execution. The following would be interesting to gather, and potentially others as we think of them.

Copied from #60315, we can start with the following:

Telemetry questions	Why	PR
What is the maximum number of alerts a single rule execution created?	Knowing these values will indicate what scale of alerts we are dealing with and decide if we need to guardrail against such large values.	#130479
What is the maximum number of actions a rule has scheduled during a single execution?	Knowing these values will indicate what scale of actions we are dealing with and decide if we need to guardrail against such large values. Note that there is a relationship between number of alerts and actions, but it's variable as some alerts could be muted, throttled, and have more than one action defined	#128891

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

@pmuellr I created this telemetry issue for 8.1 to collect data on potential guardrails. If you feel there are other areas we should collect, can you add them to the table above?

This issue overlaps a bit with #113465 but gathers scope outside of long running rules.

[mikecote]

Prior conversation on capturing the max number of alerts created during execution => #116047.

[pmuellr]

side discussion with Mike: we're thinking capturing percentiles rather than min/max. For instance, we can easily compare p50 vs p90, and determine if the value at p90 is an outlier vs p50, if it's much larger than p50. If they're similar, the p50 is probably not an outlier. We're less interested in outliers than things that are more consistently "not good".

[gmmorris]

stretch: How much event loop delay was occurring during a rule execution?

I'm wondering if we can split this issue into several deliverables (no need for multiple issues, just multiple PRs).
I'm worried about the stretch causing us to miss the 8.1FF date.
If in doubt, prioritise making incremental progress that makes the FF date.

[mikecote]

We should split this into smaller deliverables. There may be an issue already for "How much time was spent in elasticsearch searches?".

[mikecote]

Removing How much time was spent in elasticsearch searches? (moved to #125967) and How much event loop delay was occurring during a rule execution? (moved to #124366).

[mikecote]

I will leave it up to the person who picks up the issue if they want to do both telemetry questions at once or if they feel it should be split up and create a separate issue for one of the other telemetry questions.

[mikecote]

Might be worth using percentiles and maybe breaking down by rule type.