TemplateUpgradeService get stuck in repeatedly upgrading templates after upgrade to 5.6.0

[imotov]

It manifests itself by the following messages that might linger in the log files for a while after upgrade if x-pack is installed.

[2017-09-13T13:24:57,961][INFO ][o.e.c.m.TemplateUpgradeService] [node1] Starting template upgrade to version 5.6.0, 2 templates will be updated and 0 will be removed 
[2017-09-13T13:24:57,985][INFO ][o.e.c.m.TemplateUpgradeService] [node1] Finished upgrading templates to version 5.6.0 

The problem is occurring because during application of templates the order of elements in the template mapping can get shuffled causing the follow-up check if update is need to fail. I am working on the fix.

[admlko]

I'm still getting this with 6.2.1:

[2018-02-01T10:09:09,146][INFO ][o.e.c.m.TemplateUpgradeService] [analyzer01] Starting template upgrade to version 6.1.2, 1 templates will be updated and 0 will be removed
[2018-02-01T10:09:09,260][INFO ][o.e.c.m.TemplateUpgradeService] [analyzer01] Finished upgrading templates to version 6.1.2
[2018-02-01T10:09:18,168][INFO ][o.e.c.m.TemplateUpgradeService] [analyzer01] Starting template upgrade to version 6.1.2, 1 templates will be updated and 0 will be removed
[2018-02-01T10:09:18,277][INFO ][o.e.c.m.TemplateUpgradeService] [analyzer01] Finished upgrading templates to version 6.1.2

[estolfo]

I am encountering this issue when running the rest-api YAML tests with the Ruby client in Docker.

The behavior observed: The rest api tests were passing when I ran elasticsearch and the tests outside docker. But when I ran both elasticsearch and the tests in docker, ES stopped responding halfway through the tests. I thought maybe Docker/ES were running out of memory. Here is an example of the error on Jenkins. I've put the error in this gist as well, in case the Jenkins job is no longer available when this issue is investigated.

After inspecting the pending_tasks queue and correlating the tasks with parts of the Elasticsearch codebase, I found that the TemplateUpgradeService was running in between each test when we were deleting all index templates. The code run between each test was: $client.indices.delete_template(name: '*')

I’m guessing the cause of the issue is that the TemplateUpgradeService can get itself into a deadlock if called too often. The only way I was able to resolve this was to not call delete_template with name: '*' and instead delete specific templates in between tests.

Please let me know if there's any other information you need to investigate or if I can help reproduce the issue.