[esArchiver] unable to load newly captured archive of .kibana

[spalger]

It seems that something about the v2 migrations is preventing newly saved archives from being loaded correctly. It seems the aliases saved by the esArchiver are breaking the migrations so we probably need to exclude some or all of the aliases for .kibana indexes. Thoughts @rudolf?

To reproduce run the functional tests in one tab:

node scripts/functional_tests_server --config test/functional/config.js

Then save an archive in another tab once the server is running:

node scripts/es_archiver save foobar .kibana --config test/functional/config.js

Finally, try to load the archive and see the error:

node scripts/es_archiver load foobar --config test/functional/config.js

The error produced by esArchiver is:

$ node scripts/es_archiver.js load foobar --config test/functional/config.js
 info [foobar] Loading "mappings.json"
 info [foobar] Loading "data.json.gz"
 info [foobar] Deleted existing index ".kibana_8.0.0_001"
 info [foobar] Created index ".kibana_1"
 info [foobar] Indexed 1 docs into ".kibana_1"
ERROR UNHANDLED ERROR
ERROR Error: Request failed with status code 500
          at createError (/Users/spalger/kbn-dev/master/kibana/node_modules/axios/lib/core/createError.js:16:15)
          at settle (/Users/spalger/kbn-dev/master/kibana/node_modules/axios/lib/core/settle.js:17:12)
          at IncomingMessage.handleStreamEnd (/Users/spalger/kbn-dev/master/kibana/node_modules/axios/lib/adapters/http.js:260:11)
          at IncomingMessage.emit (events.js:327:22)
          at endReadableNT (internal/streams/readable.js:1327:12)
          at processTicksAndRejections (internal/process/task_queues.js:80:21)

And the error produced by es/kibana is:

   │ info [o.e.c.m.MetadataCreateIndexService] [spalger-imac.local] [.kibana_1] creating index, cause [api], templates [], shards [1]/[0]
   │ info [o.e.c.r.a.AllocationService] [spalger-imac.local] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_1][0]]"
   │ info [o.e.c.m.MetadataMappingService] [spalger-imac.local] [.kibana_1/kc2KYTIXTN2oAef1J_Bxkg] update_mapping [_doc]
   │ proc [kibana] server    log   [13:53:28.879] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH
   │ proc [kibana] server    log   [13:53:28.882] [error][savedobjects-service] [.kibana] [index_not_found_exception]: no such index [.kibana_8.0.0_001]
   │ proc [kibana] server    log   [13:53:28.883] [error][savedobjects-service] [.kibana] migration failed, dumping execution log:
   │ proc [kibana] server    log   [13:53:28.883] [info][savedobjects-service] [.kibana] INIT RESPONSE
   │ proc [kibana] server    log   [13:53:28.883] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH
   │ proc [kibana] server    log   [13:53:28.884] [error][http] Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. ResponseError: index_not_found_exception
   │ proc [kibana]     at migrationStateActionMachine (/Users/spalger/kbn-dev/master/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.ts:160:11)
   │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
   │ proc [kibana]     at async Promise.all (index 0)
   │ proc [kibana]     at /Users/spalger/kbn-dev/master/kibana/src/core/server/saved_objects/routes/migrate.ts:27:7
   │ proc [kibana]     at /Users/spalger/kbn-dev/master/kibana/src/core/server/saved_objects/routes/utils.ts:74:14
   │ proc [kibana]     at Router.handle (/Users/spalger/kbn-dev/master/kibana/src/core/server/http/router/router.ts:272:30)
   │ proc [kibana]     at handler (/Users/spalger/kbn-dev/master/kibana/src/core/server/http/router/router.ts:227:11)
   │ proc [kibana]     at exports.Manager.execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
   │ proc [kibana]     at Object.internals.handler (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
   │ proc [kibana]     at exports.execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
   │ proc [kibana]     at Request._lifecycle (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
   │ proc [kibana]     at Request._execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)
   │ proc [kibana] server   error  [13:53:28.872]  Error: Internal Server Error
   │ proc [kibana]     at HapiResponseAdapter.toInternalError (/Users/spalger/kbn-dev/master/kibana/src/core/server/http/router/response_adapter.ts:49:19)
   │ proc [kibana]     at Router.handle (/Users/spalger/kbn-dev/master/kibana/src/core/server/http/router/router.ts:286:34)
   │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
   │ proc [kibana]     at handler (/Users/spalger/kbn-dev/master/kibana/src/core/server/http/router/router.ts:227:11)
   │ proc [kibana]     at exports.Manager.execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
   │ proc [kibana]     at Object.internals.handler (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
   │ proc [kibana]     at exports.execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
   │ proc [kibana]     at Request._lifecycle (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
   │ proc [kibana]     at Request._execute (/Users/spalger/kbn-dev/master/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

esArchiver rewrites the index name https://github.com/elastic/kibana/blob/master/packages/kbn-es-archiver/src/lib/indices/generate_index_records_stream.ts#L51 into a v1 migration index name, but still keeps the v2 migration alias (.kibana_8.0.0) which causes the migration to fail.

[tylersmalley]

@spalger, think this can be closed at this point? We're getting close to being able to block esArchiver entirely from interacting with .kibana indices once #102552 is completed.

[spalger]

I'm not opposed