Handle overloaded agent's heartbeat timeout

[Andyz26]

Context

We found a case where agents with shutdown workers got leaked:

• A job with multiple workers is being shut down.
• Some workers receive the shutdown signals and terminate the running workers. However, some workers fail to receive the signals (timeout, etc.) and continue to be alive. Now, all upstream traffic is re-distributed to these workers, and they become very busy (peaked CPU/network usage).
• Retried kill signals cannot reach these workers while the worker's host agent's heartbeat message to the control plane timed out, causing them to be leaked.

Improvements:

• increase heartbeat connection client's thread priority
• increase heartbeat connection timeout settings.

Checklist

• ./gradlew build compiles code correctly
• Added new tests where applicable
• ./gradlew test passes all tests
• Extended README or added javadocs where applicable

[sundargates]

should we change the defaults for everyone? Can we just change this in our codebase?

[Andyz26]

see the other comment.

[sundargates]

same here.

[sundargates]

It feels strange that we send heartbeats every 10 seconds by default but wait for 90 seconds for those heartbeats to be acknowledged. I think if you want to change this, you should also change the interval.

[Andyz26]

In a happy case where the request is completed right away nothing changes. I think the semantics of hb every 10 seconds is good, but we want to wait longer before aborting the request to avoid the leak here when both client and the control-plane are actually healthy (in this case, the request can be completed if the connection waits long enough).

[github-actions]

Test Results

615 tests  ±0   605 ✅ ±0   8m 9s ⏱️ +6s
142 suites ±0    10 💤 ±0 
142 files   ±0     0 ❌ ±0 

Results for commit 418a0d0. ± Comparison against base commit f9d3a5c.

♻️ This comment has been updated with latest results.