ILM - Optionally short circuit min_age for empty indices

[jakelandis]

There are some conditions that may result in ILM managing empty indexes. For example, a Beat that was set up to use ILM with an age based rollover (i.e. every 1 day), but (for what ever reason) that Beat stops sending data. ILM will keep rolling over that index based on that age (i.e. every day) creating an empty index per time period (i.e. an empty index per day).

Normally the delete phase has an associated "min_age" that allows the index to stay around for the given period of time. In the case where ILM ends up managing an empty index, it may delay deleting an empty index.

For example, if the ILM policy rolls over empty indexes every day and there is a 90 day "min_age" for the delete phase, there would be up to 90 empty indexes.

This enhancement request is to optionally allow a phase to short-circuit the "min_age" if there are no documents in the index.

Something like:

"delete": {
        "min_age": "90d",
        "ignore_min_age_if_index_empty" : true,
        "actions": {
          "delete": {}
        }
      }

This could also be applied generally for the other phases too, such that an empty index would go through the phases very quickly.

[elasticmachine]

Pinging @elastic/es-core-features

[jasontedor]

For example, a Beat that was set up to use ILM with an age based rollover (i.e. every 1 day), but (for what ever reason) that Beat stops sending data. ILM will keep rolling over that index based on that age (i.e. every day) creating an empty index per time period (i.e. an empty index per day).

For example, if the ILM policy rolls over empty indexes every day and there is a 90 day "min_age" for the delete phase, there would be up to 90 empty indexes.

This doesn't feel right to me, introducing more functional surface to document and not break, and code for us to maintain for what can be argued is a configuration error.

[jakelandis]

We discussed this today and decided to close this request. This is edge case and other then a single occurrence we have not heard of others having this issue.

[drewwats]

I ended up here hoping for the same. Would be nice to have something like below. In general, I don't need ILM to run on empty indexes. Seems to just create clutter.

"warm": {
"min_age": "8d",
"ignore_empty_index": "true"
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"warm_data": "true"
}
},
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 50
},
"shrink": {
"number_of_shards": 1
}
}
}

[KannappanSomu]

same problem , due to this we are hitting max shards limit in the cluster as empty indexes are managed by ILM and retained for 2 months period . Please add this feature

[probakowski]

As we (Data Management team) decided that we will add min_* conditions to rollover I'll reopen this issue to track progress.