Breaking changes for 8.0

[odacremolbap]

List of breaking changes we plan to do for 8.0.0. All items must have an owner.

Metricbeat

• kubernetes, state_container, metric kubernetes.container.id being replaced with container.runtime and container.id. @MichaelKatsoulis @ChrsMark
  [Metricbeat]Kubernetes: make state_container metricset ECS compliant #13884
• kubernetes, state_container, metric kubernetes.container.name being replaced by container.name and kubernetes.container.name being replaced by container.image.name @MichaelKatsoulis @ChrsMark
  Add container image and container name ECS fields for state_container #23802
• Remove deprecated fields from docker module. @mtojek Docker: remove deprecated fields #27933
• Remove deprecated fields from kafka module. @jsoriano Remove deprecated kafka fields #27938
• Remove deprecated fields from kubernetes module. @MichaelKatsoulis @ChrsMark
• Remove deprecated fields from nats module. @endorama
• Remove deprecated fields from redis module. @aspacca Redis: remove deprecated fields #28246
• Refactor jolokia fields (under discussion)
  Refactor Jolokia metrics to provide dynamic mapping #13316 @jsoriano
• Remove default_region in aws module config
  [Metricbeat] Remove default_region in aws module  #13535
  @kaiyan-sheng
• Remove network and diskio metrics from ec2 metricset
  [Metricbeat] Remove network and diskio metrics from ec2 metricset #13359
  @kaiyan-sheng
• k8s Metadata TODOs for v8.0.0
  k8s Metadata TODOs for v8.0.0 #16483
  @ChrsMark @MichaelKatsoulis
• Remove aws_partition from AWS
  Remove aws_partition config parameter for AWS #23590
  @kaiyan-sheng
• Renaming GCP metric names
  [Metricbeat] gcp: fields do not respect naming best practices #27231
  @endorama

Filebeat

• Use the flattened type to store attributes from mongodb json logs, and remove log.original (see [Filebeat] [MongoDB] Support MongoDB 4.4 json logs #24774 (comment) for context). (TBD if this is even possible with the current license) @jsoriano
• Change validation on Filebeat AWS S3 input so that if both var.bucket_arn and var.queue_url are not set Filebeat will exit with an error
  Filebeat: Have filesets disabled unless explicitly configured #27526
  @aspacca
• Make all filesets disabled by default. @adriansr
  [Filebeat][proposal] Disable filesets that are not configured #17256
  Filebeat: Make all filesets disabled in default configuration #27762
  Filebeat: Have filesets disabled unless explicitly configured #27526
• Remove deprecated coredns fields @efd6
  Remove deprecated coredns fields for 8.0 #28196

Libbeat

• Remove agent.hostname field from libbeat
  Remove agent.hostname field from libbeat #18328
  Module dashboards should not use non-ECS agent.hostname fields #12983 (remove the alias) @kaiyan-sheng

• Make default_field: false the default for fields.
  Make default_field: false the default for v8.0 #28215

[odacremolbap]

Items to consider:

• [metricbeat] Remove deprecated fields #11835
• Module dashboards should not use non-ECS agent.hostname fields #12983

[ChrsMark]

Also this might be related: #13316

cc: @jsoriano, @exekias

[kaiyan-sheng]

• Remove default_region in aws module config: [Metricbeat] Remove default_region in aws module  #13535

• Remove network and diskio metrics from ec2 metricset: [Metricbeat] Remove network and diskio metrics from ec2 metricset #13359

[jsoriano]

Also this might be related: #13316

Added to the list so we don't forget, thanks @ChrsMark!

@kaiyan-sheng I have added the topics you mention to the list.

[aspacca]

#27612 (comment)

Change validation on Filebeat AWS S3 input so that if both var.bucket_arn and var.queue_url are not set Filebeat will exit with an error (changed in 7.15 waiting for #27526)

#27526 (comment)
Currently not explicitly disabled filesets are validated and started, we could switch this behaviour to validate and start only explicitly enabled filesets

[P1llus]

Removing old module aliases:
Observability: #27919
Security: #27915

[adriansr]

Filebeat: Make all filesets disabled by default. @adriansr
#17256
#27762
#27526

[adriansr]

Added:

• Make default_field: false the default for fields.
  Make default_field: false the default for v8.0 #28215

[narph]

for @narph
#20981 - azure eventhub renaming and mapping of fields
#20936 , #28284 - rename of the storage metricset for consistency
#28282 remove deprecated configuration in perfmon metricset