[Stack Monitoring] Align logstash's package mappings

[klacabane]

Summary

Align the logstash package mappings with the metricbeat ones (.monitoring-logstash-mb).

Steps to get the deltas:

• bump the package kibana.version to ^8.5.0
• start an elastic-package with version 8.5.0 elastic-package stack up -v --version 8.5.0-SNAPSHOT
• clone the diff tool https://github.com/klacabane/metrics-mappings-to-monitoring and npm install
• run a diff on the kibana package npm run print-diff --package logstash --use-local. The --use-local is a temporary workaround until [Stack Monitoring] Cleanup unused mappings properties elasticsearch#88899 is merged and available in snapshot. The diff tool includes a local version of the patched metricbeat mappings so that we don't print outdated deltas

Interpreting delta

Because the collection modes differs some differences in the mappings are expected:

• +data_stream.* - only appears in agent-collected data
• +event.(agent_id_status|ingested) - only appears in agent-collected data, defined by .fleet_agent_id_verification-1 component template
• -metricset.* - only appears in metricbeat-collected data

Besides these properties any other delta should be patched:

• for ecs properties like event.* or service.* every metric data streams' ecs.yml of the package should be updated. be careful not to patch the logs data streams as these will be handled separately
• for metricset-specific deltas like indices_stats or kibana_stats, only the corresponding data stream package-fields.yml should be updated. It can be found by searching for the term in the corresponding metricbeat module as the field will be documented in either the reference document or the fields in the _meta folder

See documentation to update packages locally

AC

• when running the diff for logstash mappings, only the expected delta are showing up
• Stack Monitoring's Logstash views are all powered by agent-collected metrics

[klacabane]

Mappings alignment blocked by elastic/kibana#138596

[klacabane]

Not a blocker anymore since there is a way to define nested properties, still look like a bug elastic/kibana#138596 (comment)