Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] How to migrate to node roles from node attrs. Closes #65855 #71160

Merged
merged 13 commits into from
Apr 27, 2021

Conversation

debadair
Copy link
Contributor

@debadair debadair commented Mar 31, 2021

Closes #65855

Preview: https://elasticsearch_71160.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/migrate-index-allocation-filters.html

In addition to the basic steps outlined here:

  • Are there upgrade/order implications that need to be called out, other than you should complete the upgrade before modifying the node roles, policies, and templates?

  • What is necessary on cloud deployments wrt template changes? Do they need to remove the old default template?

  • Are there any interactions with autoscaling or restores that need to be called out?

@debadair debadair added >docs General docs changes :Data Management/ILM+SLM Index and Snapshot lifecycle management v8.0.0 v7.10.0 v7.11.0 v7.12.0 v7.13.0 labels Mar 31, 2021
@debadair debadair marked this pull request as ready for review April 5, 2021 22:35
@elasticmachine elasticmachine added Team:Data Management Meta label for data/management team Team:Docs Meta label for docs team labels Apr 5, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

Copy link
Contributor

@andreidan andreidan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @debadair

I think this is a good start for the guide. I've left a few comments.

We should elaborate on the motivations behind needing these steps, either inline or maybe in a new troubleshooting section?

I'm still working on summarising the scenarios and proposed solutions and will comment here or send them over a gdoc when finished

@Leaf-Lin
Copy link
Contributor

Leaf-Lin commented Apr 9, 2021

Can we also include explicit settings and examples that require change?
For example,

  • allocate uses classic allocation filter index.routing.allocation.require.<some_node_attribute> was the setting to be removed. When one migrate to node roles, all node attributes should be migrated to node roles.
    For example, you can see the following index setting were required to be allocated on the hot (attribute) nodes:
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "require" : {
              "data" : "hot"
            }
          }
        }
      }
    },

This should be converted to include routing with _tier_preference:

        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot"
            }
          }

  • migrate leverages the new "data tiers" support in 7.10+, and uses data tier allocation filtering (i.e. index.routing.allocation.include._tier_preference). This works using node roles

  • ILM automatically injects the migrate action in the warm and cold phases if no allocation options are specified with the allocate action. This will still only apply the role based filtering, not the attribute based filtering.

  • You might have to manually remove the template .cloud-hot-warm-allocation-0 to avoid any index routing conflicts.

Copy link
Contributor

@andreidan andreidan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @debadair

This is looking great!

I've left just a few more minor questions and suggestions.

Copy link
Contributor

@andreidan andreidan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (assuming the docs-checks need some work to get them passing?), thanks for iterating on this Deb

This will be greatly appreciated by our users!

Comment on lines 38 to 39
When you add a data tier to an {ess} deployment,
new nodes are automatically assigned to the tier.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only true for the hot tier (data_hot node role). New indices are always allocated to the hot (or content) tier irrespective of the user adding the warm or cold tier. Shall we be more specific here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point I was trying to make was that when you add a tier to your deployment, nodes are configured with the corresponding role automatically. I tweaked the wording to make that clearer.

@debadair debadair merged commit 633e132 into elastic:master Apr 27, 2021
debadair added a commit to debadair/elasticsearch that referenced this pull request Apr 27, 2021
…5855 (elastic#71160)

* [DOCS] Document how to migrate to node roles from node attrs. Closes elastic#65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit to debadair/elasticsearch that referenced this pull request Apr 27, 2021
…5855 (elastic#71160)

* [DOCS] Document how to migrate to node roles from node attrs. Closes elastic#65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit to debadair/elasticsearch that referenced this pull request Apr 27, 2021
…5855 (elastic#71160)

* [DOCS] Document how to migrate to node roles from node attrs. Closes elastic#65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit to debadair/elasticsearch that referenced this pull request Apr 27, 2021
…5855 (elastic#71160)

* [DOCS] Document how to migrate to node roles from node attrs. Closes elastic#65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit to debadair/elasticsearch that referenced this pull request Apr 27, 2021
…5855 (elastic#71160)

* [DOCS] Document how to migrate to node roles from node attrs. Closes elastic#65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit that referenced this pull request Apr 27, 2021
…71160) (#72346)

* [DOCS] Document how to migrate to node roles from node attrs. Closes #65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit that referenced this pull request Apr 27, 2021
…71160) (#72345)

* [DOCS] Document how to migrate to node roles from node attrs. Closes #65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit that referenced this pull request Apr 27, 2021
…71160) (#72344)

* [DOCS] Document how to migrate to node roles from node attrs. Closes #65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit that referenced this pull request Apr 27, 2021
…71160) (#72343)

* [DOCS] Document how to migrate to node roles from node attrs. Closes #65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
debadair added a commit that referenced this pull request Apr 27, 2021
…71160) (#72342)

* [DOCS] Document how to migrate to node roles from node attrs. Closes #65855

* [DOCS] Incorporated review comments

* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >docs General docs changes Team:Data Management Meta label for data/management team Team:Docs Meta label for docs team v7.10.0 v7.11.0 v7.12.0 v7.13.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migration guide from node attributes to node roles for hot/warm/cold deployments
5 participants