[APM] Agent configuration should indicate whether it was applied remotely or not

[sorenlouv]

Currently a user cannot determine if a configuration was successfully applied on an agent or not. They might also want to know how many of their agents have the latest config, and how many are still on an older (perhaps local) config.

The problem of stable agent ids
The initial discussions stalled because agents doesn't have a concept of persistent ids (that are stable across restarts). This means that given two status reports with the first saying config was not applied and the second saying the config was applied we cannot determine if there is one agent that had a problem, restarted and then successfully applied the config; or if there are two agents where one couldn't apply the config and the other could.

Simplification
If we cut some corners we might not need stable agent ids initially. Instead of the UI displaying exactly which agents have the most recent configuration, we will be able to display whether a configuration was applied or not. This will not tell the user whether their entire fleet was updated, but at least they know that a non-zero amount of agents have been updated.

Implementation
APM Server queries elasticsearch for configurations via Kibana on behalf of agents. Agents already send the id of the currently applied configuration as etag. The etag (configuration id) is currently created in APM Server but going forward it should be created in Kibana. The etag can either be a hash of the configuration contents, or simply a randomly generated uuid. The important thing is that it changes every time the config changes.
Whenever Kibana receives an etag it will find the matching configuration and set the applied field to true to indicate that the config has been applied.

TODO

• Add field etag to configuration and update it every time the configuration changes
• Add field applied to configuration which is initially false (Migration? https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html)
• Change field applied to true whenever APM Server queries for a config and the config matches the etag sent by APM Server
• Reset field applied to false whenever configuration changes

[elasticmachine]

Pinging @elastic/apm-ui

[katrin-freihofner]

Would Successfully applied: 2 minutes ago be an additional column per configuration (row)? Please let me know how I can help with this.

[sorenlouv]

Talking to @jalvz we ended up with not having a timestamp, and just having a boolean flag on each configuration that indicates whether said configuration was applied or not.
I'll update the description.

[sorenlouv]

Please let me know how I can help with this.

I think from a design perspective all we need is to show the status for each configuration (one for each row in the table). We can use the EUI health component:
https://elastic.github.io/eui/#/display/health

[jalvz]

is this done?

[sorenlouv]

Fixed in: #46995