[Fleet] Cleanup empty indices

[ruflin]

I have a cluster running for a few years now and looking at the last year there are lots of empty indicies. The majority of these empty indices come from Fleet / data stream naming schema. Below is just an example but there are hundreds more:

green  open metrics-endpoint.metadata_current_default                             xL7UXVP-RRaZxrkGVzEoCA 1 0          0     0    225b    225b
yellow open .ds-metrics-system.filesystem-default-2021.04.19-000009               CvBT6zzURwWxyrpZXA6ygQ 1 1      86418     0  15.3mb  15.3mb
yellow open .ds-synthetics-icmp-default-2022.02.27-000010                         Zmrg8a-ST6y5CRNILnfHzA 1 1          0     0    225b    225b
yellow open .ds-logs-generic-default-2022.05.12-000015                            OWBdQnaPSf6UsIMSGYKnqw 1 1          0     0    225b    225b
yellow open .ds-metrics-elastic_agent.filebeat-default-2022.06.08-000019          qyZYHZk4R-qt2vKSHKvPSQ 1 1          0     0    225b    225b
yellow open .ds-metrics-elastic.agent.endpoint-security-default-2022.03.15-000020 J4eSxJZGTgODumrOtKDFgA 1 1          0     0    225b    225b
yellow open .ds-metrics-system.load-default-2022.01.31-000019                     Mo-YH99oR3CpoUCcDFuPgg 1 1          0     0    225b    225b
yellow open .ds-logs-elastic.agent-default-2021.04.19-000009                      l8HENUTTRjWwmWpLyBzgcQ 1 1          0     0    225b    225b

I think many of these empty indices come from rollovers that had to be done based on time or changes in mappings. I was looking for an easy way to cleanup these indices and stumbled over elastic/elasticsearch#73349 for ILM in Elasticsearch and elastic/curator#1263 in curator.

On my end, I want to keep the old data but get rid of all the empty indices. I have not found a fully automated way yet to do this.

The discussion I would like to have here is, is there any value in keeping these empty indices in the first place? Should a data stream automatically take care of these empty indices and wipe then? I filed this in Kibana / Fleet because the rollovers are likely triggered by Fleet in many cases but I could well see that this is a problem that should be tackled by Elasticsearch.

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[ruflin]

I learned there are some ongoing efforts on the Elasticsearch side to help with this problem: elastic/elasticsearch#86203 Also some changes on the ILM side are already implemented: elastic/elasticsearch#83345 Both are part of elastic/elasticsearch#89283

[ruflin]

I had a quick chat with @dakrone about this issue and he came up with a neat idea specific to the Fleet issue on how to resolve it: elastic/elasticsearch#89346 The nice part about his proposal is that it would not only solve the issue but also make package upgrades more efficient I think as no rollovers have to triggered and creating empty shards. @joshdover @kpollich FYI