Upgrade assistant keeps looping on resume reindex step if Kibana restarts

[bhavyarm]

Kibana version: 7.17 BC2

Elasticsearch version: 7.17 BC2

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: If the user is in the middle of reindexing and Kibana restarts - upgrade assistant displays reindex paused but clicking on resume reindex step - just keeps looping and displays reindex-paused - resume reindex step again.

Please note this is happening on shakespeare (index) dataset from 6.8.x.

Steps to reproduce:

1. Start your 7.17 ES/Kibana with 6.8.20 data - I am slacking the data folder. Its too big for the github issue
2. Go to upgrade assistant - click on reindexing in ES panel (I clicked on reindexing on indexes from top of the list)
3. Restart Kibana
4. Upgrade assistant displays reindex paused for shakespeare index pattern

5. Click on reindex paused - > resume reindexing ->accept changes

6. Reindex starts but pauses at 65% and UA UI says resume reindexing again

7. If you open index management - reindexed-shakespeare index exists but in yellow state

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[bhavyarm]

Network call -

[alisonelizabeth]

Still looking into this... but leaving my initial findings here:

It appears there is an issue with re-associating the credentials used to process the reindex task on restart. The credentials are undefined when the user tries to resume the reindex, which causes the loop. You can find the related code starting here. There is a "pause window", and eventually this expires and the paused message appears in the UI again.

I still haven't pinpointed the exact cause and/or when this regression might have occurred. There were changes made to this area of the code when we added support for API keys (#111451), although I think this is unrelated. We also added support for bulk reindexing ~2 years ago, which touched this area of the code. This wasn't heavily tested, so it's possible it was introduced then.

Also, note - in order to reproduce, I had to restart Kibana when the reindex process was on step 3, "Reindex documents".

[alisonelizabeth]

PR open - #123727

[alisonelizabeth]

Fixed via #123727