-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* [DOCS] Adds redirect for Standard Upgrade doc * [DOCS] Updates Upgrade docs * [DOCS] Incorporates review comments * [DOCS] Adds info on saved objects * [DOCS] Fixes broken link * [DOCS] Fixes placement of section on unknown saved objects * [DOCS] Removes duplicate topic (cherry picked from commit a13dafc)
- Loading branch information
Showing
9 changed files
with
261 additions
and
247 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,31 +1,72 @@ | ||
[[upgrade]] | ||
== Upgrade {kib} | ||
|
||
To upgrade from 7.16.0 or earlier to {version}, | ||
To upgrade from 7.16.0 or earlier to {version}, | ||
**you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to | ||
{stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade]. | ||
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*. | ||
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*. | ||
|
||
Rolling upgrades are unsupported in {kib}. To upgrade, | ||
Rolling upgrades are unsupported in {kib}. To upgrade, | ||
you must shut down all {kib} instances, install the new software, and restart {kib}. | ||
Upgrading while older {kib} instances are running can cause data loss or upgrade failures. | ||
|
||
[WARNING] | ||
==== | ||
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>. | ||
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>. | ||
In case of an upgrade failure, you can roll back to an | ||
earlier version of {kib}. To roll back, you **must** have a | ||
{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature | ||
state. By default, snapshots include the `kibana` feature state. | ||
==== | ||
|
||
For more information about upgrading, | ||
For more information about upgrading, | ||
refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.] | ||
|
||
IMPORTANT: You can upgrade to pre-release versions for testing, | ||
IMPORTANT: You can upgrade to pre-release versions for testing, | ||
but upgrading from a pre-release to the General Available version is unsupported. | ||
You should use pre-release versions only for testing in a temporary environment. | ||
|
||
include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1] | ||
[float] | ||
=== Upgrading multiple {kib} instances | ||
When upgrading several {kib} instances connected to the same {es} cluster, | ||
ensure that all outdated instances are shut down before starting the upgrade. | ||
|
||
include::upgrade/logging-configuration-changes.asciidoc[] | ||
Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel, | ||
which allows all instances to participate in the upgrade migration in parallel. | ||
|
||
For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects, | ||
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to | ||
complete the upgrade migration before bringing up the remaining instances. | ||
|
||
[float] | ||
[[preventing-migration-failures]] | ||
=== Preparing for migration | ||
|
||
There are extra steps you can follow to ensure you are ready for migration. | ||
|
||
[float] | ||
==== Ensure your {es} cluster is healthy | ||
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has: | ||
|
||
* Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices | ||
* Sufficient heap size | ||
* A "green" cluster status | ||
|
||
[float] | ||
==== Ensure that all {kib} instances are the same | ||
When you perform an upgrade migration of different {kib} versions, the migration can fail. | ||
Ensure that all {kib} instances are running the same version, configuration, and plugins. | ||
|
||
[float] | ||
==== Back up your data | ||
Be sure to have a {ref}/snapshot-restore.html[snapshot] of all your data before attempting a migration. | ||
If something goes wrong during migration, you can restore from the snapshot and try again. | ||
|
||
Review the <<resolve-migrations-failures,common causes of {kib} upgrade failures>> and how to prevent them. | ||
|
||
|
||
include::upgrade/saved-objects-migration.asciidoc[] | ||
|
||
include::upgrade/resolving-migration-failures.asciidoc[] | ||
|
||
include::upgrade/rollback-migration.asciidoc[] |
125 changes: 125 additions & 0 deletions
125
docs/setup/upgrade/resolving-migration-failures.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,125 @@ | ||
[[resolve-migrations-failures]] | ||
=== Resolve migration failures | ||
|
||
Migrating {kib} primarily involves migrating saved object documents to be compatible | ||
with the new version. | ||
|
||
[float] | ||
==== Resolve saved object migration failures | ||
|
||
If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to | ||
perform the migration again when the process restarts. Do not delete any saved objects indices to | ||
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and | ||
later does not require deleting indices to release a failed migration lock. | ||
|
||
If upgrade migrations fail repeatedly, refer to | ||
<<preventing-migration-failures, preparing for migration>>. | ||
When you address the root cause for the migration failure, | ||
{kib} automatically retries the migration. | ||
If you're unable to resolve a failed migration, contact Support. | ||
|
||
|
||
[float] | ||
[[upgrade-migrations-old-indices]] | ||
==== Handle old `.kibana_N` indices | ||
|
||
After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). | ||
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to. | ||
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version. | ||
|
||
[float] | ||
==== Handle known issues with {fleet} beta | ||
If you see a`timeout_exception` or `receive_timeout_transport_exception` error, | ||
it might be from a known known issue in 7.12.0 if you tried the {fleet} beta. | ||
Upgrade migrations fail because of a large number of documents in the `.kibana` index, | ||
which causes {kib} to log errors such as: | ||
|
||
[source,sh] | ||
-------------------------------------------- | ||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms] | ||
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54] | ||
-------------------------------------------- | ||
|
||
For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue]. | ||
|
||
|
||
[float] | ||
==== Handle corrupt saved objects | ||
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster, | ||
especially when there are custom integrations that create saved objects in your environment. | ||
|
||
Saved objects that are corrupted through manual editing or integrations cause migration | ||
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`. | ||
For a successful upgrade migration, you must fix or delete corrupt documents. | ||
|
||
For example, you receive the following error message: | ||
|
||
[source,sh] | ||
-------------------------------------------- | ||
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index. | ||
-------------------------------------------- | ||
|
||
To delete the documents that cause migrations to fail, take the following steps: | ||
|
||
. Remove the write block which the migration system has placed on the previous index: | ||
+ | ||
[source,sh] | ||
-------------------------------------------- | ||
PUT .kibana_7.12.1_001/_settings | ||
{ | ||
"index": { | ||
"blocks.write": false | ||
} | ||
} | ||
-------------------------------------------- | ||
|
||
. Delete the corrupt document: | ||
+ | ||
[source,sh] | ||
-------------------------------------------- | ||
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275 | ||
-------------------------------------------- | ||
|
||
. Restart {kib}. | ||
+ | ||
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**. | ||
|
||
[float] | ||
[[unknown-saved-object-types]] | ||
==== Handle documents for unknown saved objects | ||
Migrations will fail if saved objects belong to an unknown | ||
saved object type. Unknown saved objects are typically caused by | ||
to the {es} index, or by disabling a plugin that had previously | ||
created a saved object. | ||
|
||
We recommend using the {kibana-ref-all}/7.17/upgrade-assistant.html[Upgrade Assistant] | ||
to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved | ||
object types will also log the following warning message: | ||
|
||
[source,sh] | ||
-------------------------------------------- | ||
CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes. | ||
-------------------------------------------- | ||
|
||
If you fail to remedy this, your upgrade to 8.0+ will fail with a message like: | ||
|
||
[source,sh] | ||
-------------------------------------------- | ||
Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index. | ||
-------------------------------------------- | ||
|
||
[float] | ||
==== Handle incompatible settings or mappings | ||
Matching index templates that specify `settings.refresh_interval` or | ||
`mappings` are known to interfere with {kib} upgrades. | ||
This can happen when index templates are defined manually. | ||
|
||
To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates. | ||
|
||
[float] | ||
==== Handle incompatible `xpack.tasks.index` configuration setting | ||
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` | ||
with the configuration setting `xpack.tasks.index: ".tasks"`, | ||
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration | ||
setting prevents upgrade migrations from starting. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
[[upgrade-migrations-rolling-back]] | ||
=== Roll back to a previous version of {kib} | ||
|
||
If you've followed <<preventing-migration-failures, preparing for migration>> | ||
and <<resolve-migrations-failures, resolving migration failures>>, and | ||
{kib} is still unable to successfully upgrade, rollback {kib} until | ||
you're able to identify and fix the root cause. | ||
|
||
WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with | ||
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}. | ||
Any changes made after an upgrade are lost when you roll back to a previous version. | ||
|
||
To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version. | ||
|
||
[float] | ||
==== Roll back by restoring a backup snapshot | ||
|
||
. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state. | ||
By default, snapshots include the `kibana` feature state. | ||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. | ||
. To delete all saved object indices, use `DELETE /.kibana*`. | ||
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot. | ||
. Start all {kib} instances on the older version you want to rollback to. | ||
|
||
[float] | ||
==== (Not recommended) Roll back without a backup snapshot | ||
|
||
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances. | ||
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state. | ||
. Delete the version-specific indices created by the failed upgrade migration. | ||
+ | ||
For example, to rollback from a failed upgrade | ||
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`. | ||
. Inspect the output of `GET /_cat/aliases`. | ||
+ | ||
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually. | ||
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. | ||
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`. | ||
. To remove the write block from the roll back indices, use | ||
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}` | ||
. Start {kib} on the older version you want to rollback to. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
[[saved-object-migrations]] | ||
=== Migrate saved objects | ||
|
||
Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version. | ||
|
||
NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*]. | ||
To access the assistant, go to *Stack Management > Upgrade Assistant*. | ||
|
||
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}. | ||
|
||
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly. | ||
|
||
[float] | ||
[[upgrade-migrations-process]] | ||
==== How saved objects migrations work | ||
|
||
Saved objects are stored in two indices: | ||
|
||
* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`. | ||
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`. | ||
|
||
The index aliases `.kibana` and `.kibana_task_manager` always point to | ||
the most up-to-date saved object indices. | ||
|
||
When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. | ||
Before you upgrade, shut down old nodes to prevent losing acknowledged writes. | ||
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later | ||
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions. | ||
|
||
.Saved object indices and aliases per {kib} version | ||
[options="header"] | ||
|======================= | ||
|Upgrading from version | Outdated index (alias) | ||
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias) | ||
| 7.4.0 through 7.11.x | ||
| `.kibana_N` (`.kibana` alias) | ||
|
||
`.kibana_task_manager_N` (`.kibana_task_manager` alias) | ||
|======================= |
Oops, something went wrong.