7.12.0 upgrade migrations fail with timeout_exception or receive_timeout_transport_exception

[rudolf]

v7.12.0 migrations can fail if the .kibana index has a large number of saved objects or the Elasticsearch cluster is under heavy load.

This will cause errors logs like:

[.kibana] [receive_timeout_transport_exception]: [instance-0000000002][10.42.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
[.kibana] [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]

The root cause is a bug in the v2 migrations that means Kibana will only wait for 60s for some of the steps. When a cluster is busy or there are a lot of fleet-agent-events _reindex or _update_by_query tasks can take longer than 60s leading to a timeout.

Workaround for large fleet-agent-events

If you've used fleet and have a large number of fleet-agent-events follow this workaround

1. First establish that there are large number of fleet-agent-events:

   POST .kibana/_search?filter_path=aggregations
   {
     "aggs": {
       "saved_object_type": {
         "terms": {"field": "type"}
       }
     }
   }
   

2. Shutdown Kibana
3. Wait for any reindex tasks started by Kibana to complete. The output of the following request shouldn't show any reindex tasks for the .kibana* indices:

   GET  _tasks?actions=*update/byquery,reindex&group_by=parents&detailed
   # Cancel any tasks where the "description" field shows it's operating on a `.kibana*` index
   # e.g. `"description": "update-by-query [.kibana_7.12.0_001]"
   POST _tasks/<task id from above>/_cancel
   # Verify that all tasks have been cancelled:
   GET  _tasks?actions=*update/byquery,reindex&group_by=parents&detailed
   

4. Delete fleet-agent-events

   POST .kibana,.kibana_7.12.0_*/_delete_by_query?conflicts=proceed&wait_for_completion=false
   {
     "query": {
       "bool": {
         "must": 
           {
             "term": {
               "type": "fleet-agent-events"
             }
           }
       }
     }
   }
   

   Check that the task returned from the above operation has completed before proceeding

   GET _tasks/<id>
   

5. Restart Kibana

PR #95305

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[hueyg]

Forgive my ignorance, but how do you carry out these POST and GET commands if you cannot run Kibana to use the Console? I assume you can use CURL, but what exactly am I carrying out these commands against? What port?

I came here from the 7.13 documentation and suffering this exact issue from a cluster that I have had around since 6.x days. The upgrades where carried out from official elascticsearch repos enabled on RedHat systems.

[pgayvallet]

but how do you carry out these POST and GET commands if you cannot run Kibana to use the Console

Any CLI or GUI client, such as curl or postman

e.g for curl

curl -XPOST "https://localhost:9200/.kibana/_search?filter_path=aggregations" -d'
{
  "aggs": {
    "saved_object_type": {
      "terms": {"field": "type"}
    }
  }'

What port?

Your configured ES port. The default is 9200

[hueyg]

but how do you carry out these POST and GET commands if you cannot run Kibana to use the Console

Any CLI or GUI client, such as curl or postman

e.g for curl

curl -XPOST "https://localhost:9200/.kibana/_search?filter_path=aggregations" -d'
{
  "aggs": {
    "saved_object_type": {
      "terms": {"field": "type"}
    }
  }'

What port?

Your configured ES port. The default is 9200

Thank you pgayvallet!

[wonderland14]

Hi @pgayvallet , may I have a sample POST for postman? I am trying to open the index, because my Kibana is not accessible because I accidentally closed the kibana_security and I think some other kibana's indices. I do research but still confused how come all indices will re-open. Sorry for my ignorance 'bout these