Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ILM don't rollover empty indices #89557

Merged
merged 21 commits into from
Sep 19, 2022

Conversation

joegallo
Copy link
Contributor

@joegallo joegallo commented Aug 23, 2022

Closes #86203, related to #89283

ILM will no longer roll over empty indices by default. This is accomplished by injecting a min_docs: 1 condition into the rollover dry run request in WaitForRolloverReadyStep.

Such a condition will only be injected if there's not already a min_docs (or min_primary_shard_docs) condition associated with the rollover, so a policy can opt out of the new behavior by adding an explicit min_docs: 0 condition to the rollover action.

This can also be disabled on a cluster-wide basis by setting indices.lifecycle.rollover.only_if_has_documents to false.

so the String and Setting versions are in the same order, likewise the
IndexLifecycle list of settings..
so that everything handling the conditions is last
This is controlled via the cluster-wide
indices.lifecycle.rollover.only_if_has_documents setting, and will
only apply if the policy doesn't already have an explicit min_docs (or
min_primary_shard_docs) condition.

That is, to disable this behavior globally, use the setting; but to
disable it on a per-policy basis, add an explicit min_docs: 0
condition.
@joegallo joegallo added >enhancement :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team v8.5.0 labels Aug 23, 2022
@elasticsearchmachine
Copy link
Collaborator

Hi @joegallo, I've created a changelog YAML for you.

@elasticsearchmachine
Copy link
Collaborator

Hi @joegallo, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

@joegallo joegallo marked this pull request as ready for review September 15, 2022 13:37
@joegallo joegallo requested a review from dakrone September 15, 2022 13:37
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I left two very minor reviews, thanks for adding this, I think it'll help a lot of people!

);
}

// visible for testing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be even better if this had javadoc, even though it's essentially private :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, b4f780a

assertTrue(indexExists(originalIndex));
}, 30, TimeUnit.SECONDS);

switch (between(0, 2)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think rather than randomize between the three potential ways to force a rollover, I'd rather explicitly do each of them. This would allow us to catch issues better in the future that might only break one of the branches since we don't test each method every CI run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion, on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does 1ea80c1 work for you, @dakrone?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me, 🚢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement release highlight Team:Data Management Meta label for data/management team v8.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prevent ILM from spuriously rolling over (many) empty indices
3 participants