[BT-504] cromwell graceful restarts

[mspector]

No description provided.

[cjllanwarne]

LGTM!

[aednichols]

Nice find!

The code absolutely does look good and it is plausible to me that it solves the problem. It would be nice to see before/after timing diagram screenshots of the same workflow to make sure we have hit the right target.

[mspector]

I ran a job that simply sleeps for 120 seconds, then requested the timing diagram ("Before the fix"). Then I restarted Cromwell with the fix, and requested the timing diagram again ("After the fix").

Before the fix:

After the fix:

In the "after the fix" image, the purple bar is "cromwell starting overhead", and lasts about 60 seconds. The maroon bar is "RunningJob", and also last about 60 seconds. It seems like the Maroon bar should have also been "RunningJob".