diff --git a/docs/reference/data-management/migrate-index-allocation-filters.asciidoc b/docs/reference/data-management/migrate-index-allocation-filters.asciidoc new file mode 100644 index 0000000000000..be6b0d2c06067 --- /dev/null +++ b/docs/reference/data-management/migrate-index-allocation-filters.asciidoc @@ -0,0 +1,163 @@ +[role="xpack"] +[[migrate-index-allocation-filters]] +== Migrate index allocation filters to node roles + +If you currently use custom node attributes and +<> to +move indices through <> in a +https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management[hot-warm-cold architecture], +we recommend that you switch to using the built-in node roles +and automatic <>. +Using node roles enables {ilm-init} to automatically +move indices between data tiers. + +NOTE: While we recommend relying on automatic data tier allocation to manage +your data in a hot-warm-cold architecture, +you can still use attribute-based allocation filters to +control shard allocation for other purposes. + +To switch to using node roles: + +. <> to the appropriate data tier. +. <> from your {ilm} policy. +. <> +on new indices. +. Update existing indices to <>. + + +[discrete] +[[assign-data-tier]] +=== Assign data nodes to a data tier + +Configure the appropriate roles for each data node to assign it to one or more +data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. +A node can also have other <>. By default, new nodes are +configured with all roles. + +When you add a data tier to an {ess} deployment, +one or more nodes are automatically configured with the corresponding role. +To explicitly change the role of a node in an {ess} deployment, use the +{cloud}/ec-api-deployment-crud.html#ec_update_a_deployment[Update deployment API]. +Replace the node's `node_type` configuration with the appropriate `node_roles`. +For example, the following configuration adds the node to the hot and content +tiers, and enables it to act as an ingest node, remote, and transform node. + +[source,yaml] +---- +"node_roles": [ + "data_hot", + "data_content", + "ingest", + "remote_cluster_client", + "transform" +], +---- + +If you are directly managing your own cluster, +configure the appropriate roles for each node in `elasticsearch.yml`. +For example, the following setting configures a node to be a data-only +node in the hot and content tiers. + +[source,yaml] +---- +node.roles [ data_hot, data_content ] +---- + +[discrete] +[[remove-custom-allocation-settings]] +=== Remove custom allocation settings from existing {ilm-init} policies + +Update the allocate action for each lifecycle phase to remove the attribute-based +allocation settings. This enables {ilm-init} to inject the +<> action into each phase +to automatically transition the indices through the data tiers. + +If the allocate action does not set the number of replicas, +remove the allocate action entirely. (An empty allocate action is invalid.) + +IMPORTANT: The policy must specify the corresponding phase for each data tier in +your architecture. Each phase must be present so {ilm-init} can inject the +migrate action to move indices through the data tiers. +If you don't need to perform any other actions, the phase can be empty. +For example, if you enable the warm and cold data tiers for a deployment, +your policy must include the hot, warm, and cold phases. + +[discrete] +[[stop-setting-custom-hot-attribute]] +=== Stop setting the custom hot attribute on new indices + +When you create a data stream, its first backing index +is now automatically assigned to `data_hot` nodes. +Similarly, when you directly create an index, it +is automatically assigned to `data_content` nodes. + +On {ess} deployments, remove the `cloud-hot-warm-allocation-0` index template +that set the hot shard allocation attribute on all indices. + +[source,console] +---- +DELETE _template/.cloud-hot-warm-allocation-0 +---- +// TEST[skip:no cloud template] + +If you're using a custom index template, update it to remove the <> you used to assign new indices to the hot tier. + +[discrete] +[[set-tier-preference]] +=== Set a tier preference for existing indices. + +{ilm-init} automatically transitions managed indices through the available +data tiers by automatically injecting a <> +into each phase. + +To enable {ilm-init} to move an _existing_ managed index +through the data tiers, update the index settings to: + +. Remove the custom allocation filter by setting it to `null`. +. Set the <>. + +For example, if your old template set the `data` attribute to `hot` +to allocate shards to the hot tier, set the `data` attribute to `null` +and set the `_tier_preference` to `data_hot`. + +//// +[source,console] +---- +PUT /my-index + +PUT /my-index/_settings +{ + "index.routing.allocation.require.data": "hot" +} +---- +//// + +[source,console] +---- +PUT my-index/_settings +{ + "index.routing.allocation.require.data": null, + "index.routing.allocation.include._tier_preference": "data_hot" +} +---- +// TEST[continued] + +For indices that have already transitioned out of the hot phase, +the tier preference should include the appropriate fallback tiers +to ensure index shards can be allocated if the preferred tier +is unavailable. +For example, specify the hot tier as the fallback for indices +already in the warm phase. + +[source,console] +---- +PUT my-index/_settings +{ + "index.routing.allocation.require.data": null, + "index.routing.allocation.include._tier_preference": "data_warm,data_hot" +} +---- +// TEST[continued] + +If an index is already in the cold phase, include the cold, warm, and hot tiers. diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index 550608916d8f2..73007c3674c18 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -5,23 +5,23 @@ [partintro] -- -You can configure {ilm} ({ilm-init}) policies to automatically manage indices -according to your performance, resiliency, and retention requirements. +You can configure {ilm} ({ilm-init}) policies to automatically manage indices +according to your performance, resiliency, and retention requirements. For example, you could use {ilm-init} to: * Spin up a new index when an index reaches a certain size or number of documents * Create a new index each day, week, or month and archive previous ones * Delete stale indices to enforce data retention standards - + You can create and manage index lifecycle policies through {kib} Management or the {ilm-init} APIs. -When you enable {ilm} for {beats} or the {ls} {es} output plugin, +When you enable {ilm} for {beats} or the {ls} {es} output plugin, default policies are configured automatically. [role="screenshot"] image:images/ilm/index-lifecycle-policies.png[] [TIP] -To automatically back up your indices and manage snapshots, +To automatically back up your indices and manage snapshots, use <>. * <> @@ -29,6 +29,7 @@ use < * <> * <> * <> +* <> * <> * <> * <> @@ -50,6 +51,8 @@ include::ilm-actions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] +include::../data-management/migrate-index-allocation-filters.asciidoc[] + include::error-handling.asciidoc[] include::start-stop.asciidoc[]