[SO migration] fail the migration if unknown types are encountered #118300

pgayvallet · 2021-11-11T09:32:48Z

Summary

Fix #107678 by manually reverting #105213

Revert SO migration behavior to what it was before #105213 by having it fail fast when unknown SO types are encountered during the CHECK_UNKNOWN_DOCUMENTS step

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Release Note

Encountering unknown saved object types during a version migration will now cause Kibana to fail the upgrade. To proceed with the migration, it will be required to either re-enable the plugins owning them, or to delete the documents from the previous version's index.

…rt-on-unknown-so-types

pgayvallet · 2021-11-12T06:43:47Z

@elasticmachine merge upstream

pgayvallet · 2021-11-12T08:43:02Z

retest

pgayvallet · 2021-11-12T08:48:07Z

@elasticmachine merge upstream

pgayvallet

self-review

pgayvallet · 2021-11-12T10:26:32Z

src/core/server/saved_objects/migrations/core/migrate_raw_docs.ts

-      // Do not transform documents of unknown types
-      if (raw?._source?.type && !knownTypes.has(raw._source.type)) {
-        processedDocs.push(raw);
-      } else if (serializer.isRawSavedObject(raw, options)) {


This part of the revert was not strictly necessary, but it felt safer to get back to the behavior where migrating unknown doc fails, to be consistent with our strategy to fail the migration in such case (even if going that far should never occur)

pgayvallet · 2021-11-12T10:28:14Z

src/core/server/saved_objects/migrations/integration_tests/batch_size_bytes.test.ts

    {
-      oss: true,
+      oss: false,


Some test datasets contains objects of types registered from xpack plugins, I had to enable xpack for those tests to avoid re-generating the dataset, as the tests were failing due to the behavior reintroduced in this PR.

pgayvallet · 2021-11-12T10:28:55Z

src/core/server/saved_objects/migrations/integration_tests/batch_size_bytes.test.ts

@@ -122,7 +122,7 @@ describe('migration v2', () => {
        expect(
          records.find((rec) =>
            rec.message.startsWith(
-              `Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715275 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.`
+              `Unable to complete saved object migrations for the [.kibana] index: The document with _id "canvas-workpad-template:workpad-template-061d7868-2b4e-4dc8-8bf7-3772b52926e5" is 1715274 bytes which exceeds the configured maximum batch size of 1015275 bytes. To proceed, please increase the 'migrations.maxBatchSizeBytes' Kibana configuration option and ensure that the Elasticsearch 'http.max_content_length' configuration option is set to an equal or larger value.`


I have NO idea why, but enabling xpack changed the size of the document by one byte 🤷

An extra newline maybe or some tiny change between node versions?

elasticmachine · 2021-11-12T10:29:10Z

Pinging @elastic/kibana-core (Team:Core)

…rt-on-unknown-so-types

TinaHeiligers

Revert LGTM

TinaHeiligers · 2021-11-15T17:03:37Z

@pgayvallet shouldn't "process" be "proceed" in the release note? i.e.

To proceed with the migration, it will be required to either re-enable the plugins owning them, or to delete the documents from the previous version's index.

or

To process the migration, it will be required to either re-enable the plugins owning them, or to delete the documents from the previous version's index.

pgayvallet · 2021-11-16T09:38:06Z

shouldn't "process" be "proceed" in the release note?

Definitely should, thanks.

…rt-on-unknown-so-types

pgayvallet · 2021-11-16T11:14:00Z

@elasticmachine merge upstream

kibana-ci · 2021-11-16T12:43:23Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #6931 succeeded ef6b335
💚 Build #6642 succeeded ef6b335
💚 Build #6464 succeeded ad4a3ff
💔 Build #6461 failed 112e95a
💚 Build #6283 succeeded 55f00b5
💔 Build #6237 failed 63a1932

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#118300) * manually revert elastic#105213 * enabled xpack to register types for some IT tests * fiz doc size Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-11-16T12:57:54Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

…118300) (#118690) * manually revert #105213 * enabled xpack to register types for some IT tests * fiz doc size Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Pierre Gayvallet <pierre.gayvallet@elastic.co>

…lastic#118300) * manually revert elastic#105213 * enabled xpack to register types for some IT tests * fiz doc size Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…118300) * manually revert #105213 * enabled xpack to register types for some IT tests * fiz doc size Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…lastic#118300) * manually revert elastic#105213 * enabled xpack to register types for some IT tests * fiz doc size Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

manually revert elastic#105213

686e66e

pgayvallet added project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 v8.1.0 release_note:breaking labels Nov 11, 2021

pgayvallet added 3 commits November 11, 2021 15:05

Merge remote-tracking branch 'upstream/main' into kbn-107678-fail-sta…

b65fa72

…rt-on-unknown-so-types

enabled xpack to register types for some IT tests

63a1932

fiz doc size

55f00b5

Merge branch 'main' into kbn-107678-fail-start-on-unknown-so-types

112e95a

elastic deleted a comment from kibanamachine Nov 12, 2021

Merge branch 'main' into kbn-107678-fail-start-on-unknown-so-types

ad4a3ff

pgayvallet commented Nov 12, 2021

View reviewed changes

pgayvallet marked this pull request as ready for review November 12, 2021 10:29

pgayvallet requested a review from a team as a code owner November 12, 2021 10:29

Merge remote-tracking branch 'upstream/main' into kbn-107678-fail-sta…

ef6b335

…rt-on-unknown-so-types

TinaHeiligers approved these changes Nov 15, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/main' into kbn-107678-fail-sta…

2e3c8e4

…rt-on-unknown-so-types

Merge branch 'main' into kbn-107678-fail-start-on-unknown-so-types

a2fb3c6

pgayvallet added the auto-backport Deprecated - use backport:version if exact versions are needed label Nov 16, 2021

pgayvallet merged commit e08c633 into elastic:main Nov 16, 2021

kibanamachine mentioned this pull request Nov 16, 2021

[8.0] [SO migration] fail the migration if unknown types are encountered (#118300) #118690

Merged

This was referenced Dec 9, 2021

[Reporting] Problems with non-uniform enablement #120995

Closed

[Reporting] Ensure uniform enablement of the plugin #121020

Closed

rudolf mentioned this pull request Feb 10, 2022

Improve docs for unknown saved object types #125253

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SO migration] fail the migration if unknown types are encountered #118300

[SO migration] fail the migration if unknown types are encountered #118300

pgayvallet commented Nov 11, 2021 •

edited

Loading

pgayvallet commented Nov 12, 2021

pgayvallet commented Nov 12, 2021

pgayvallet commented Nov 12, 2021

pgayvallet left a comment

pgayvallet Nov 12, 2021

pgayvallet Nov 12, 2021

pgayvallet Nov 12, 2021

TinaHeiligers Nov 15, 2021

elasticmachine commented Nov 12, 2021

TinaHeiligers left a comment

TinaHeiligers commented Nov 15, 2021

pgayvallet commented Nov 16, 2021

pgayvallet commented Nov 16, 2021

kibana-ci commented Nov 16, 2021

kibanamachine commented Nov 16, 2021

[SO migration] fail the migration if unknown types are encountered #118300

[SO migration] fail the migration if unknown types are encountered #118300

Conversation

pgayvallet commented Nov 11, 2021 • edited Loading

Summary

Checklist

Release Note

pgayvallet commented Nov 12, 2021

pgayvallet commented Nov 12, 2021

pgayvallet commented Nov 12, 2021

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Nov 12, 2021

Choose a reason for hiding this comment

pgayvallet Nov 12, 2021

Choose a reason for hiding this comment

pgayvallet Nov 12, 2021

Choose a reason for hiding this comment

TinaHeiligers Nov 15, 2021

Choose a reason for hiding this comment

elasticmachine commented Nov 12, 2021

TinaHeiligers left a comment

Choose a reason for hiding this comment

TinaHeiligers commented Nov 15, 2021

pgayvallet commented Nov 16, 2021

pgayvallet commented Nov 16, 2021

kibana-ci commented Nov 16, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Nov 16, 2021

💚 Backport successful

pgayvallet commented Nov 11, 2021 •

edited

Loading