Manual rollover on ILM indices leads to Error

[fkelbert]

ES 7.2.0

Manually rolling over an ILM index leads to the error shown in the screenshot.

Steps to reproduce:

1. Create ILM policy with rollover after 10 docs
2. Create first ILM index (-00001) in accordance with documentation, i.e., specify is_write_index
3. Add some documents
4. Wait for first auto-rollover
5. Add some more documents
6. Manual rollover
7. Add some more documents
8. Wait for auto rollover

[elasticmachine]

Pinging @elastic/es-core-features

[dakrone]

@fkelbert can you explain what you'd want ES to do here? In this case, it went to roll over the index but it's not actually the correct index since it was manually rolled over.

Do you have a use case for manual and automatic rollover at the same time?

[gwbrown]

We discussed this in the core/features sync today.

Because Rollover information is recorded in the index metadata when an index is rolled over, we can detect this situation in ILM, and have ILM skip trying to roll over the index if it has been rolled over via the alias configured in index.lifecycle.rollover_alias. We had previously discussed this, but were concerned that it might be "too magic" because the rollover info stored in the index metadata is not easily visible anywhere and there is no way (other than the general Elasticsearch logs) to expose why ILM skipped rollover in this case. However, we could not think of any use cases where a manual rollover is performed and the case where users want to roll over again, specifically via ILM, using the same alias.

The upshot of this is that we're going to move ahead with implementing this feature to have ILM skip the rollover action if it detects that an index has been manually rolled over via the alias defined in index.lifecycle.rollover_alias, but if anyone has any objections, please do bring them up here.

[fkelbert]

The use case / situation here was as follows:

1. Create an index including corresponding index template and ILM
2. Index data
3. Realize that the index mappings within the index template need modification
4. Modify index template
5. Run rollover (_roller) to force the creation of a new index that uses the new mappings
6. Expect ILM to work just as it did before (which failed)

Is there a possibility to make the "ILM skip rollover behaviour" configurable?

Should we have handled the above situation differently?

[dakrone]

Is there a possibility to make the "ILM skip rollover behaviour" configurable?

Are you talking about the proposed rollover skip behavior? Where we would automatically detect that you manually rolled over and skip the ILM-level rollover?

Should we have handled the above situation differently?

No that sounds like a valid use case, thanks for explaining it. This will help us figure out the best solution for this issue.

[fkelbert]

Is there a possibility to make the "ILM skip rollover behaviour" configurable?

Are you talking about the proposed rollover skip behavior? Where we would automatically detect that you manually rolled over and skip the ILM-level rollover?

Yes.

Should we have handled the above situation differently?

No that sounds like a valid use case, thanks for explaining it. This will help us figure out the best solution for this issue.

Great! happy to provide more input if needed.

[dakrone]

Is there a possibility to make the "ILM skip rollover behaviour" configurable?

Are you talking about the proposed rollover skip behavior? Where we would automatically detect that you manually rolled over and skip the ILM-level rollover?

Yes.

We could probably add a setting to enable/disable the auto-rollover detection/skipping.

@gwbrown what do you think?

[gwbrown]

It's definitely possible to add a setting to control that behavior, my only concern is that I think the setting is going to be rather confusing to explain, and entirely unnecessary for most users. That's probably solvable with an info box in the docs that says something to the effect of "This is an expert setting, you probably don't need it" though.

[lio-p]

One workaround is to move the index in error to a stable state:

POST _ilm/move/<index_name>
{
  "current_step": {
    "phase": "hot",
    "action": "rollover",
    "name": "ERROR"
  },
  "next_step": {
    "phase": "hot",
    "action": "complete",
    "name": "complete"
  }
}

[fkelbert]

Thank you very much for fixing @andreidan