diff --git a/docs/developer/architecture/core/index.asciidoc b/docs/developer/architecture/core/index.asciidoc index 53720a593d3f2d..6c70205c3590d1 100644 --- a/docs/developer/architecture/core/index.asciidoc +++ b/docs/developer/architecture/core/index.asciidoc @@ -36,9 +36,3 @@ The services that core provides are: * <> * <> * <> - - - - - - diff --git a/docs/setup/upgrade/logging-configuration-changes.asciidoc b/docs/developer/architecture/core/logging-configuration-migration.asciidoc similarity index 100% rename from docs/setup/upgrade/logging-configuration-changes.asciidoc rename to docs/developer/architecture/core/logging-configuration-migration.asciidoc diff --git a/docs/developer/architecture/index.asciidoc b/docs/developer/architecture/index.asciidoc index 774292f513f038..90a0972d65f2fc 100644 --- a/docs/developer/architecture/index.asciidoc +++ b/docs/developer/architecture/index.asciidoc @@ -40,6 +40,8 @@ include::core/http-service.asciidoc[leveloffset=+1] include::core/logging-service.asciidoc[leveloffset=+1] +include::core/logging-configuration-migration.asciidoc[leveloffset=+1] + include::core/saved-objects-service.asciidoc[leveloffset=+1] include::core/uisettings-service.asciidoc[leveloffset=+1] diff --git a/docs/redirects.asciidoc b/docs/redirects.asciidoc index 6a34076f10988f..2c9d48813781aa 100644 --- a/docs/redirects.asciidoc +++ b/docs/redirects.asciidoc @@ -386,16 +386,16 @@ This content has moved. Refer to <>. This content has moved. Refer to <>. -[role="exclude",id="logging-configuration-changes"] -== Logging configuration changes - -This content has moved. Refer to <>. - [role="exclude",id="upgrade-migrations"] == Upgrade migrations This content has moved. Refer to <>. +[role="exclude",id="upgrade-standard"] +== Standard Upgrade + +This content has moved. Refer to {stack-ref}/upgrading-kibana.html[Upgrade Kibana]. + [role="exclude",id="upgrade-assistant"] == Upgrade Assistant diff --git a/docs/setup/upgrade.asciidoc b/docs/setup/upgrade.asciidoc index 98713e75d24f6a..3069d78cc692e7 100644 --- a/docs/setup/upgrade.asciidoc +++ b/docs/setup/upgrade.asciidoc @@ -1,31 +1,72 @@ [[upgrade]] == Upgrade {kib} -To upgrade from 7.16.0 or earlier to {version}, +To upgrade from 7.16.0 or earlier to {version}, **you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to {stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade]. -Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*. +Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*. -Rolling upgrades are unsupported in {kib}. To upgrade, +Rolling upgrades are unsupported in {kib}. To upgrade, you must shut down all {kib} instances, install the new software, and restart {kib}. Upgrading while older {kib} instances are running can cause data loss or upgrade failures. [WARNING] ==== -When required, {kib} automatically migrates <>. +When required, {kib} automatically migrates <>. In case of an upgrade failure, you can roll back to an earlier version of {kib}. To roll back, you **must** have a {ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state. ==== -For more information about upgrading, +For more information about upgrading, refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.] -IMPORTANT: You can upgrade to pre-release versions for testing, +IMPORTANT: You can upgrade to pre-release versions for testing, but upgrading from a pre-release to the General Available version is unsupported. You should use pre-release versions only for testing in a temporary environment. -include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1] +[float] +=== Upgrading multiple {kib} instances +When upgrading several {kib} instances connected to the same {es} cluster, +ensure that all outdated instances are shut down before starting the upgrade. -include::upgrade/logging-configuration-changes.asciidoc[] +Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel, +which allows all instances to participate in the upgrade migration in parallel. + +For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects, +you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to +complete the upgrade migration before bringing up the remaining instances. + +[float] +[[preventing-migration-failures]] +=== Preparing for migration + +There are extra steps you can follow to ensure you are ready for migration. + +[float] +==== Ensure your {es} cluster is healthy +Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has: + + * Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices + * Sufficient heap size + * A "green" cluster status + +[float] +==== Ensure that all {kib} instances are the same +When you perform an upgrade migration of different {kib} versions, the migration can fail. +Ensure that all {kib} instances are running the same version, configuration, and plugins. + +[float] +==== Back up your data +Be sure to have a {ref}/snapshot-restore.html[snapshot] of all your data before attempting a migration. +If something goes wrong during migration, you can restore from the snapshot and try again. + +Review the <> and how to prevent them. + + +include::upgrade/saved-objects-migration.asciidoc[] + +include::upgrade/resolving-migration-failures.asciidoc[] + +include::upgrade/rollback-migration.asciidoc[] diff --git a/docs/setup/upgrade/resolving-migration-failures.asciidoc b/docs/setup/upgrade/resolving-migration-failures.asciidoc new file mode 100644 index 00000000000000..454dfe948fe4e8 --- /dev/null +++ b/docs/setup/upgrade/resolving-migration-failures.asciidoc @@ -0,0 +1,125 @@ +[[resolve-migrations-failures]] +=== Resolve migration failures + +Migrating {kib} primarily involves migrating saved object documents to be compatible +with the new version. + +[float] +==== Resolve saved object migration failures + +If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to +perform the migration again when the process restarts. Do not delete any saved objects indices to +to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and +later does not require deleting indices to release a failed migration lock. + +If upgrade migrations fail repeatedly, refer to +<>. +When you address the root cause for the migration failure, +{kib} automatically retries the migration. +If you're unable to resolve a failed migration, contact Support. + + +[float] +[[upgrade-migrations-old-indices]] +==== Handle old `.kibana_N` indices + +After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). +{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to. +The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version. + +[float] +==== Handle known issues with {fleet} beta +If you see a`timeout_exception` or `receive_timeout_transport_exception` error, +it might be from a known known issue in 7.12.0 if you tried the {fleet} beta. +Upgrade migrations fail because of a large number of documents in the `.kibana` index, +which causes {kib} to log errors such as: + +[source,sh] +-------------------------------------------- +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] + +Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] +-------------------------------------------- + +For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue]. + + +[float] +==== Handle corrupt saved objects +To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster, +especially when there are custom integrations that create saved objects in your environment. + +Saved objects that are corrupted through manual editing or integrations cause migration +failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`. +For a successful upgrade migration, you must fix or delete corrupt documents. + +For example, you receive the following error message: + +[source,sh] +-------------------------------------------- +Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. +-------------------------------------------- + +To delete the documents that cause migrations to fail, take the following steps: + +. Remove the write block which the migration system has placed on the previous index: ++ +[source,sh] +-------------------------------------------- +PUT .kibana_7.12.1_001/_settings +{ + "index": { + "blocks.write": false + } +} +-------------------------------------------- + +. Delete the corrupt document: ++ +[source,sh] +-------------------------------------------- +DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 +-------------------------------------------- + +. Restart {kib}. ++ +The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**. + +[float] +[[unknown-saved-object-types]] +==== Handle documents for unknown saved objects +Migrations will fail if saved objects belong to an unknown +saved object type. Unknown saved objects are typically caused by +to the {es} index, or by disabling a plugin that had previously +created a saved object. + +We recommend using the {kibana-ref-all}/7.17/upgrade-assistant.html[Upgrade Assistant] +to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved +object types will also log the following warning message: + +[source,sh] +-------------------------------------------- +CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes. +-------------------------------------------- + +If you fail to remedy this, your upgrade to 8.0+ will fail with a message like: + +[source,sh] +-------------------------------------------- +Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index. +-------------------------------------------- + +[float] +==== Handle incompatible settings or mappings +Matching index templates that specify `settings.refresh_interval` or +`mappings` are known to interfere with {kib} upgrades. +This can happen when index templates are defined manually. + +To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates. + +[float] +==== Handle incompatible `xpack.tasks.index` configuration setting +In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` +with the configuration setting `xpack.tasks.index: ".tasks"`, +upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration +setting prevents upgrade migrations from starting. diff --git a/docs/setup/upgrade/rollback-migration.asciidoc b/docs/setup/upgrade/rollback-migration.asciidoc new file mode 100644 index 00000000000000..1b87d0f335b8c0 --- /dev/null +++ b/docs/setup/upgrade/rollback-migration.asciidoc @@ -0,0 +1,41 @@ +[[upgrade-migrations-rolling-back]] +=== Roll back to a previous version of {kib} + +If you've followed <> +and <>, and +{kib} is still unable to successfully upgrade, rollback {kib} until +you're able to identify and fix the root cause. + +WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with +your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}. +Any changes made after an upgrade are lost when you roll back to a previous version. + +To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version. + +[float] +==== Roll back by restoring a backup snapshot + +. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state. + By default, snapshots include the `kibana` feature state. +. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. +. To delete all saved object indices, use `DELETE /.kibana*`. +. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot. +. Start all {kib} instances on the older version you want to rollback to. + +[float] +==== (Not recommended) Roll back without a backup snapshot + +. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. +. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state. +. Delete the version-specific indices created by the failed upgrade migration. ++ +For example, to rollback from a failed upgrade +to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`. +. Inspect the output of `GET /_cat/aliases`. ++ +If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually. +Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. +For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`. +. To remove the write block from the roll back indices, use +`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}` +. Start {kib} on the older version you want to rollback to. diff --git a/docs/setup/upgrade/saved-objects-migration.asciidoc b/docs/setup/upgrade/saved-objects-migration.asciidoc new file mode 100644 index 00000000000000..cc4406f8cdd1f6 --- /dev/null +++ b/docs/setup/upgrade/saved-objects-migration.asciidoc @@ -0,0 +1,39 @@ +[[saved-object-migrations]] +=== Migrate saved objects + +Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <> are compatible with the new version. + +NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*]. +To access the assistant, go to *Stack Management > Upgrade Assistant*. + +WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}. + +WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly. + +[float] +[[upgrade-migrations-process]] +==== How saved objects migrations work + +Saved objects are stored in two indices: + +* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`. +* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`. + +The index aliases `.kibana` and `.kibana_task_manager` always point to +the most up-to-date saved object indices. + +When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. +Before you upgrade, shut down old nodes to prevent losing acknowledged writes. +To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later +adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions. + +.Saved object indices and aliases per {kib} version +[options="header"] +|======================= +|Upgrading from version | Outdated index (alias) +| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias) +| 7.4.0 through 7.11.x +| `.kibana_N` (`.kibana` alias) + +`.kibana_task_manager_N` (`.kibana_task_manager` alias) +|======================= diff --git a/docs/setup/upgrade/upgrade-migrations.asciidoc b/docs/setup/upgrade/upgrade-migrations.asciidoc deleted file mode 100644 index f75251089bf558..00000000000000 --- a/docs/setup/upgrade/upgrade-migrations.asciidoc +++ /dev/null @@ -1,228 +0,0 @@ -[float] -[[saved-object-migrations]] -=== Saved object migrations - -Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <> are compatible with the new version. - -NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*]. -To access the assistant, go to *Stack Management > Upgrade Assistant*. - -WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}. - -WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly. - -[float] -[[upgrade-migrations-process]] -==== Background - -Saved objects are stored in two indices: - -* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`. -* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`. - -The index aliases `.kibana` and `.kibana_task_manager` always point to -the most up-to-date saved object indices. - -When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. -Before you upgrade, shut down old nodes to prevent losing acknowledged writes. -To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later -adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions. - -.Saved object indices and aliases per {kib} version -[options="header"] -|======================= -|Upgrading from version | Outdated index (alias) - -`.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias) -| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias) -| 7.4.0 through 7.11.x -| `.kibana_N` (`.kibana` alias) - -`.kibana_task_manager_N` (`.kibana_task_manager` alias) -|======================= - -==== Upgrading multiple {kib} instances -When upgrading several {kib} instances connected to the same {es} cluster, -ensure that all outdated instances are shut down before starting the upgrade. - -Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel, -which allows all instances to participate in the upgrade migration in parallel. - -For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects, -you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to -complete the upgrade migration before bringing up the remaining instances. - -[float] -[[preventing-migration-failures]] -==== Preventing migration failures -Review the common causes of {kib} upgrade failures and how to prevent them. - -[float] -====== Documents found for unknown saved object types -Migrations will fail if saved objects belong to an unknown -saved object type. Unknown saved objects are typically caused by -to the {es} index, or by disabling a plugin that had previously -created a saved object. - -We recommend using the https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[Upgrade Assistant] -to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved -object types will also log the following warning message: - -[source,sh] --------------------------------------------- -CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes. --------------------------------------------- - -If you fail to remedy this, your upgrade to 8.0+ will fail with a message like: -[source,sh] --------------------------------------------- -Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index. --------------------------------------------- - -[float] -===== timeout_exception or receive_timeout_transport_exception -There is a known issue in 7.12.0 for users who tried the {fleet} beta. -Upgrade migrations fail because of a large number of documents in the `.kibana` index, which causes {kib} to log errors such as: - - -[source,sh] --------------------------------------------- -Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] - -Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] --------------------------------------------- - -For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue]. - -[float] -===== Corrupt saved objects -To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster, -especially when there are custom integrations that create saved objects in your environment. - -Saved objects that are corrupted through manual editing or integrations cause migration -failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`. -For a successful upgrade migration, you must fix or delete corrupt documents. - -For example, you receive the following error message: - -[source,sh] --------------------------------------------- -Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. --------------------------------------------- - -To delete the documents that cause migrations to fail, take the following steps: - -. Remove the write block which the migration system has placed on the previous index: -+ -[source,sh] --------------------------------------------- -PUT .kibana_7.12.1_001/_settings -{ - "index": { - "blocks.write": false - } -} --------------------------------------------- - -. Delete the corrupt document: -+ -[source,sh] --------------------------------------------- -DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 --------------------------------------------- - -. Restart {kib}. -+ -The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**. - -Be sure you have a snapshot before you delete the corrupt document. If you are unable to restore from a snapshot, it is recommended to also delete the `temp` and `target` indices the migration creates before you restart {kib} and retry the snapshot restore. - -[float] -===== User defined index templates that cause new `.kibana*` indices to have incompatible settings or mappings -Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades. - -To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates. - -[float] -===== An unhealthy {es} cluster -Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has: - - * Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices - * Sufficient heap size - * A "green" cluster status - -[float] -===== Different versions of {kib} connected to the same {es} index -When you perform an upgrade migration of different {kib} versions, the migration can fail. -Ensure that all {kib} instances are running the same version, configuration, and plugins. - -[float] -===== Incompatible `xpack.tasks.index` configuration setting -In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`, -upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting. - -[float] -[[resolve-migrations-failures]] -==== Resolving migration failures - -If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to -perform the migration again when the process restarts. Do not delete any saved objects indices to -to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and -later does not require deleting indices to release a failed migration lock. - -If upgrade migrations fail repeatedly, refer to -<>. -When you address the root cause for the migration failure, -{kib} automatically retries the migration. -If you're unable to resolve a failed migration, contact Support. - -[float] -[[upgrade-migrations-rolling-back]] -==== Rolling back to a previous version of {kib} - -If you've followed <> -and <>, and -{kib} is still unable to successfully upgrade, rollback {kib} until -you're able to identify and fix the root cause. - -WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with -your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}. -Any changes made after an upgrade are lost when you roll back to a previous version. - -To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version. - -[float] -===== Roll back by restoring a backup snapshot - -. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state. - By default, snapshots include the `kibana` feature state. -. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. -. To delete all saved object indices, use `DELETE /.kibana*`. -. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot. -. Start all {kib} instances on the older version you want to rollback to. - -[float] -===== (Not recommended) Roll back without a backup snapshot - -. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. -. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state. -. Delete the version-specific indices created by the failed upgrade migration. -+ -For example, to rollback from a failed upgrade -to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`. -. Inspect the output of `GET /_cat/aliases`. -+ -If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually. -Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. -For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`. -. To remove the write block from the roll back indices, use -`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}` -. Start {kib} on the older version you want to rollback to. - -[float] -[[upgrade-migrations-old-indices]] -==== Handling old `.kibana_N` indices - -After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). -{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to. -The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.