[8.0] [DOCS] Updates the Upgrade docs (#125288)

docs/developer/architecture/core/index.asciidoc

@@ -36,9 +36,3 @@ The services that core provides are:
 * <<logging-service, Logging service>>
 * <<saved-objects-service, Saved Objects service>>
 * <<ui-settings-service, UI settings service>>
-
-
-
-
-
-

docs/developer/architecture/index.asciidoc

@@ -40,6 +40,8 @@ include::core/http-service.asciidoc[leveloffset=+1]
 
 include::core/logging-service.asciidoc[leveloffset=+1]
 
+include::core/logging-configuration-migration.asciidoc[leveloffset=+1]
+
 include::core/saved-objects-service.asciidoc[leveloffset=+1]
 
 include::core/uisettings-service.asciidoc[leveloffset=+1]

docs/redirects.asciidoc

@@ -386,16 +386,16 @@ This content has moved. Refer to <<managing-data-views>>.
 
 This content has moved. Refer to <<kibana-role-management>>.
 
-[role="exclude",id="logging-configuration-changes"]
-== Logging configuration changes
-
-This content has moved. Refer to <<logging-config-changes>>.
-
 [role="exclude",id="upgrade-migrations"]
 == Upgrade migrations
 
 This content has moved. Refer to <<saved-object-migrations>>.
 
+[role="exclude",id="upgrade-standard"]
+== Standard Upgrade
+
+This content has moved. Refer to {stack-ref}/upgrading-kibana.html[Upgrade Kibana].
+
 [role="exclude",id="upgrade-assistant"]
 == Upgrade Assistant
 

docs/setup/upgrade.asciidoc

@@ -1,31 +1,72 @@
 [[upgrade]]
 == Upgrade {kib}
 
-To upgrade from 7.16.0 or earlier to {version}, 
+To upgrade from 7.16.0 or earlier to {version},
 **you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to
 {stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade].
-Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*. 
+Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.
 
-Rolling upgrades are unsupported in {kib}. To upgrade,  
+Rolling upgrades are unsupported in {kib}. To upgrade,
 you must shut down all {kib} instances, install the new software, and restart {kib}.
 Upgrading while older {kib} instances are running can cause data loss or upgrade failures.
 
 [WARNING]
 ====
-When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>. 
+When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
 In case of an upgrade failure, you can roll back to an
 earlier version of {kib}. To roll back, you **must** have a
 {ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
 state. By default, snapshots include the `kibana` feature state.
 ====
 
-For more information about upgrading, 
+For more information about upgrading,
 refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.]
 
-IMPORTANT: You can upgrade to pre-release versions for testing, 
+IMPORTANT: You can upgrade to pre-release versions for testing,
 but upgrading from a pre-release to the General Available version is unsupported.
 You should use pre-release versions only for testing in a temporary environment.
 
-include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1]
+[float]
+=== Upgrading multiple {kib} instances
+When upgrading several {kib} instances connected to the same {es} cluster,
+ensure that all outdated instances are shut down before starting the upgrade.
 
-include::upgrade/logging-configuration-changes.asciidoc[]
+Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel,
+which allows all instances to participate in the upgrade migration in parallel.
+
+For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
+you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
+complete the upgrade migration before bringing up the remaining instances.
+
+[float]
+[[preventing-migration-failures]]
+=== Preparing for migration
+
+There are extra steps you can follow to ensure you are ready for migration.
+
+[float]
+==== Ensure your {es} cluster is healthy
+Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:
+
+ * Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
+ * Sufficient heap size
+ * A "green" cluster status
+
+[float]
+==== Ensure that all {kib} instances are the same
+When you perform an upgrade migration of different {kib} versions, the migration can fail.
+Ensure that all {kib} instances are running the same version, configuration, and plugins.
+
+[float]
+==== Back up your data
+Be sure to have a {ref}/snapshot-restore.html[snapshot] of all your data before attempting a migration.
+If something goes wrong during migration, you can restore from the snapshot and try again.
+
+Review the <<resolve-migrations-failures,common causes of {kib} upgrade failures>> and how to prevent them.
+
+
+include::upgrade/saved-objects-migration.asciidoc[]
+
+include::upgrade/resolving-migration-failures.asciidoc[]
+
+include::upgrade/rollback-migration.asciidoc[]

docs/setup/upgrade/resolving-migration-failures.asciidoc

@@ -0,0 +1,125 @@
+[[resolve-migrations-failures]]
+=== Resolve migration failures
+
+Migrating {kib} primarily involves migrating saved object documents to be compatible
+with the new version.
+
+[float]
+==== Resolve saved object migration failures
+
+If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
+perform the migration again when the process restarts. Do not delete any saved objects indices to
+to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
+later does not require deleting indices to release a failed migration lock.
+
+If upgrade migrations fail repeatedly, refer to
+<<preventing-migration-failures, preparing for migration>>.
+When you address the root cause for the migration failure,
+{kib} automatically retries the migration.
+If you're unable to resolve a failed migration, contact Support.
+
+
+[float]
+[[upgrade-migrations-old-indices]]
+==== Handle old `.kibana_N` indices
+
+After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
+{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
+The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
+
+[float]
+==== Handle known issues with {fleet} beta
+If you see a`timeout_exception` or `receive_timeout_transport_exception` error,
+it might be from a known known issue in 7.12.0 if you tried the {fleet} beta.
+Upgrade migrations fail because of a large number of documents in the `.kibana` index,
+which causes {kib} to log errors such as:
+
+[source,sh]
+--------------------------------------------
+Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
+
+Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
+--------------------------------------------
+
+For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].
+
+
+[float]
+==== Handle corrupt saved objects
+To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster,
+especially when there are custom integrations that create saved objects in your environment.
+
+Saved objects that are corrupted through manual editing or integrations cause migration
+failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
+For a successful upgrade migration, you must fix or delete corrupt documents.
+
+For example, you receive the following error message:
+
+[source,sh]
+--------------------------------------------
+Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
+--------------------------------------------
+
+To delete the documents that cause migrations to fail, take the following steps:
+
+. Remove the write block which the migration system has placed on the previous index:
++
+[source,sh]
+--------------------------------------------
+PUT .kibana_7.12.1_001/_settings
+{
+  "index": {
+    "blocks.write": false
+  }
+}
+--------------------------------------------
+
+. Delete the corrupt document:
++
+[source,sh]
+--------------------------------------------
+DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
+--------------------------------------------
+
+. Restart {kib}.
++
+The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.
+
+[float]
+[[unknown-saved-object-types]]
+==== Handle documents for unknown saved objects
+Migrations will fail if saved objects belong to an unknown
+saved object type. Unknown saved objects are typically caused by
+to the {es} index, or by disabling a plugin that had previously
+created a saved object.
+
+We recommend using the {kibana-ref-all}/7.17/upgrade-assistant.html[Upgrade Assistant] 
+to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved
+object types will also log the following warning message:
+
+[source,sh]
+--------------------------------------------
+CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes.
+--------------------------------------------
+
+If you fail to remedy this, your upgrade to 8.0+ will fail with a message like:
+
+[source,sh]
+--------------------------------------------
+Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index.
+--------------------------------------------
+
+[float]
+==== Handle incompatible settings or mappings
+Matching index templates that specify `settings.refresh_interval` or
+`mappings` are known to interfere with {kib} upgrades.
+This can happen when index templates are defined manually.
+
+To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.
+
+[float]
+==== Handle incompatible `xpack.tasks.index` configuration setting
+In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks`
+with the configuration setting `xpack.tasks.index: ".tasks"`,
+upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration
+setting prevents upgrade migrations from starting.

docs/setup/upgrade/rollback-migration.asciidoc

@@ -0,0 +1,41 @@
+[[upgrade-migrations-rolling-back]]
+=== Roll back to a previous version of {kib}
+
+If you've followed <<preventing-migration-failures, preparing for migration>>
+and <<resolve-migrations-failures, resolving migration failures>>, and
+{kib} is still unable to successfully upgrade, rollback {kib} until
+you're able to identify and fix the root cause.
+
+WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
+your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
+Any changes made after an upgrade are lost when you roll back to a previous version.
+
+To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.
+
+[float]
+==== Roll back by restoring a backup snapshot
+
+. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
+   By default, snapshots include the `kibana` feature state.
+. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
+. To delete all saved object indices, use `DELETE /.kibana*`.
+. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
+. Start all {kib} instances on the older version you want to rollback to.
+
+[float]
+==== (Not recommended) Roll back without a backup snapshot
+
+. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
+. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
+. Delete the version-specific indices created by the failed upgrade migration.
++
+For example, to rollback from a failed upgrade
+to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
+. Inspect the output of `GET /_cat/aliases`.
++
+If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
+Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
+For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
+. To remove the write block from the roll back indices, use
+`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
+. Start {kib} on the older version you want to rollback to.

docs/setup/upgrade/saved-objects-migration.asciidoc

@@ -0,0 +1,39 @@
+[[saved-object-migrations]]
+=== Migrate saved objects
+
+Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
+
+NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*].
+To access the assistant, go to *Stack Management > Upgrade Assistant*.
+
+WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.
+
+WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
+
+[float]
+[[upgrade-migrations-process]]
+==== How saved objects migrations work
+
+Saved objects are stored in two indices:
+
+* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
+* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.
+
+The index aliases `.kibana` and `.kibana_task_manager` always point to
+the most up-to-date saved object indices.
+
+When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
+Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
+To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
+adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.
+
+.Saved object indices and aliases per {kib} version
+[options="header"]
+|=======================
+|Upgrading from version | Outdated index (alias)
+| 6.5.0 through 7.3.x    | `.kibana_N` (`.kibana` alias)
+| 7.4.0 through 7.11.x
+| `.kibana_N` (`.kibana` alias)
+
+`.kibana_task_manager_N` (`.kibana_task_manager` alias)
+|=======================