Automatically handle tasks that might be stuck in launching status

[ssalinas]

After implementing mesos master failover handling we've seen a few cases where a task will make it to the master but not be propagated to the agent before the master shuts down. This leaves us with a task that is launched in Singularity, but has no state on the master or agent. To compound this, with our newer faster reconnect, it's possible that we trigger our reconciliation before the master has finished building its full state, so the initial reconcile cycle may not pick up on the missed task.

This adds a step to the scheduler poller that will trigger an explicit reconciliation for any task in a launching state for more than 3 minutes (configurable). It will then trigger a re-check every 15s until it gets a status from the master. These should move the tasks to LOST eventually if they have not actually been launched. It also adds some logging to the offer cache, which we were not properly cleaning up upon master restart

Still TODO:

• fix tests
• clean up the recent reconcile timestamps so it doesn't grown forever

[pschoenfelder]

🚢