Filter to only starting tasks when reconciling

[relistan]

This should fix #2122 by filtering properly to only launching tasks. It was previously targeted all tasks that were not starting which I am pretty certain the opposite of the intent. Please review #2122 for all the details.

[ssalinas]

Thanks for the PR, my internet is out and I haven’t been able to work on much via LTE 😢

[ssalinas]

@pschoenfelder can you help make sure this makes it through the branches and gets merged?

[pschoenfelder]

👍 Will start rolling out on Monday.

[relistan]

Looks like the merge resolution screwed up the formatting and the formatter is angry (why CI failed). I can fix Monday morning.

[ssalinas]

Coming back to this, it seems to have had the opposite effect on our end. Reading the logic again if we want all tasks in a launching state and we are inspecting the the presence of a 'starting' history event in zk, the previous logic was actually correct. If the 'starting' event is not present in zk, explicitly reconcile the task because we want to see if we missed the first event back from the executor for it

This seems to point to #2122 being caused by a separate issue and tasks not actually making it from launching -> starting for some reason in your cluster

I'm going to open a PR to revert this for the moment as it is dramatically slowing down some of our internals. Feel free to stay on the older commit for the moment and we can keep looking into the cause of your issue as well @relistan