Fixes start API not setting FAILED metadata to RETRY. Also fixes bug …

[dbbaughe]

…with releaseLock not correclty releasing an updated lock

Issue #, if available:

Description of changes:
Previously Start API was setting everything to STARTED no matter what, now it will:
Skip updating any metadata that is already in: [STARTED, INIT, RETRY]
Set these to STARTED: [FINISHED, STOPPED]
And set FAILED to RETRY

There was also a bug with runRollupJob where we constantly renewed the lock which is updating the seqNo/primaryTerm. So once a job that was running fails or completes the outside releaseLock will fail because of version conflict which means then any potential next execution from a retry call needs to wait for the ttl to expire. This adds release locks inside the runRollupJob where needed, although not the cleanest solution with it littered in a few places. Will go back later and see how we can clean up that part (i.e. return lock for outside release or something).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

Merging #347 (27c8cb7) into master (423a825) will increase coverage by 0.20%.
The diff coverage is 62.50%.

@@             Coverage Diff              @@
##             master     #347      +/-   ##
============================================
+ Coverage     75.21%   75.41%   +0.20%     
- Complexity     1349     1358       +9     
============================================
  Files           183      183              
  Lines          6972     6993      +21     
  Branches       1133     1138       +5     
============================================
+ Hits           5244     5274      +30     
+ Misses         1125     1114      -11     
- Partials        603      605       +2     

Impacted Files	Coverage Δ	Complexity Δ
.../rollup/action/start/TransportStartRollupAction.kt	58.00% <56.52%> (+10.94%)	6.00 <1.00> (+3.00)
...asticsearch/indexmanagement/rollup/RollupRunner.kt	65.28% <77.77%> (-0.23%)	25.00 <2.00> (+1.00)	⬇️
...ment/rollup/action/get/TransportGetRollupAction.kt	82.35% <0.00%> (-5.89%)	3.00% <0.00%> (ø%)
...ndexstatemanagement/IndexStateManagementHistory.kt	75.96% <0.00%> (-1.93%)	27.00% <0.00%> (-1.00%)
...agement/indexstatemanagement/ManagedIndexRunner.kt	55.65% <0.00%> (+0.61%)	41.00% <0.00%> (ø%)
...anagement/indexstatemanagement/model/Transition.kt	64.61% <0.00%> (+1.53%)	5.00% <0.00%> (ø%)
...arch/indexmanagement/rollup/RollupSearchService.kt	62.00% <0.00%> (+2.00%)	9.00% <0.00%> (+1.00%)
...adata/TransportUpdateManagedIndexMetaDataAction.kt	88.88% <0.00%> (+3.70%)	10.00% <0.00%> (+1.00%)
...ch/indexmanagement/rollup/RollupMetadataService.kt	69.15% <0.00%> (+5.47%)	30.00% <0.00%> (+4.00%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 423a825...27c8cb7. Read the comment docs.