[Fleet] Add event based telemetry for agent actions

[juliaElastic]

We don't have a lot of insight currently on how much agent actions are being used, and what % of actions succeeded/failed.

It would be useful to add event based telemetry for outcome of actions:

• Data to be captured: action type, how many agents actioned, succeeded, failed, still in progress.
• Implementation options:
  • Hourly telemetry on the overall action statuses (x succeeded, x failed, x in progress) - similarly to /action_status API being called regularly to report on Agent activity UI.
  • Hourly telemetry could include the most common (top 3) error messages, to have some insight on the most common errors.
  • Event based telemetry for each agent action ack would probably be an overkill as it could potentially trigger thousands of events for bulk actions.

Telemetry would be also useful to track usage/errors of new actions: request diagnostics, remote restart.

A corresponding indexer and job will be needed to index the data in telemetry cluster.

Related:

• [Fleet] Agent activity enhancements #141206

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amitkanfer]

YES!

[juliaElastic]

@kpollich raised this issue to add telemetry on agent actions, please review

[kpollich]

Thanks, @juliaElastic - this looks like a great set of meaningful enhancements to our telemetry sender.

I think there's maybe some duplication of effort with the private issue I just linked above, though having separate telemetry more closely related to actions seems extremely useful.

I initially marked this as Needs Definition, but I think this is in a pretty workable state already. Thanks for the great description 🙂 - let's plan this for Sprint 4/8.7.