Use a scripted reindex for re-writing document _ids for shared saved object types

[joshdover]

In #86247, we decided to go with a client-side reindex to support the shared saved object type migration which requires re-writing document _ids.

If we decide to stop handling unknown saved object types (see #107678), we may be able to change this to be a scripted reindex in Elasticsearch instead to increase performance of this migration. This would help users upgrade large Kibana installations to 8.0+ by reducing the amount of downtime. It'd also help with decreasing the complexity of the v2 migration algo.

Long-term this is likely an improvement we are going to need. There have been several cases now where plugins have written 100k+ or even 1M+ documents into the .kibana index (eg. #106308, #95321). Ideally, the v2 algo could handle such cases without manual intervention while we make sure to prevent the situation from happening in the first place. We should be able to implement later in the 8.x release to benefit customers who upgrade to the next major version later in the release lifecycle.

Open questions:

• How much of an improvement would this change be to the migration process? See the profiling options in v2 migrations needs to support re-writing document _id's to support saved-objects in multiple spaces #86247 for inspiration on how we could test this.
• What's the size of this effort and will it be worth the cost?
• Are there other future migration use cases we would need to add a client-side reindex for again? If so, this is likely not worth the effort.

Blockers:

• Remove support for migrating unknown types in 8.0 (pending decision) Refuse to start with unknown saved object types in 8.0 #107678
• Remove the outdated documents step (issue currently closed, but should be reopened if we move ahead with the above) [SO Migration v2] Remove the OUTDATED_DOCUMENTS_* steps #103731
• Add UUIDv5 support in Elasticsearch's painless language

cc @jportner

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jportner]

I have two comments/concerns:

Legacy URL aliases

When a namespaceType conversion causes an object ID to change, Kibana also creates a legacy URL alias object for it with its old ID and a few other fields. Effectively this creates two documents from one. It's not clear to me if this is possible during a scripted migration.

User-defined migrations

FWIW, the current implementation does technically allow user-defined migrations to change object types and/or IDs if necessary. It sounds like switching to a scripted reindex would preclude that.

We don't have anyone writing these types of migrations yet but I have seen some examples where the idea is being toyed with.

[rudolf]

Once all saved object types have been converted the need for conversion effectively falls away, so from that point forward we could go back to using a reindex operation instead of the client-side reindex. E.g. if we convert all saved object types in 8.1 then 8.1 -> 8.2 can use a reindex, but 8.0 -> 8.2 would still use the client-side reindex. In a semver world we could remove the client-side reindex in 9.0.0, but most users could potentially benefit from better performance before that.

[pgayvallet]

Given that the _id rewrite was a one time thing, and knowing we're now only updating (reindexing) the outdated documents without performing a full index reindex, I think it should be fine to close this. Feel free to reopen otherwise.