update heartbeat checks in scheduler

[quasiben]

attempt to fix #3877

[quasiben]

I was a little confused by this statement:

I recommend modifying the distributed/scheduler.py::heartbeat_interval
function to remove the maximum

For now, I increased the maximum heartbeat to 10s

[mrocklin]

For now, I increased the maximum heartbeat to 10s

At the 20k worker level even this results in heartbeats every 500us, which might still be a burden on the scheduler. I recommend that we drop the maximum interval entirely.

[quasiben]

At the 20k worker level even this results in heartbeats every 500us, which might still be a burden on the scheduler. I recommend that we drop the maximum interval entirely.

Sorry for being dense here but if we removed the maximum what should we be doing for messages like the following when len(workers)>200

return {
            "status": "OK",
            "time": time(),
            "heartbeat-interval": heartbeat_interval(len(self.workers)),
        }

[mrocklin]

Lets say that we want no more than 200 heartbeats a second or something like that, and adjust the heartbeat interval accordingly to match that based on the number of workers.

…

On Mon, Jun 15, 2020 at 8:46 AM Benjamin Zaitlen ***@***.***> wrote: At the 20k worker level even this results in heartbeats every 500us, which might still be a burden on the scheduler. I recommend that we drop the maximum interval entirely. Sorry for being dense here but if we removed the maximum what should we be doing for messages like the following when len(workers)>200 return { "status": "OK", "time": time(), "heartbeat-interval": heartbeat_interval(len(self.workers)), } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3896 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTCA7BLTFDB2Y37DFNTRWY64ZANCNFSM4N6IQKUQ> .

[jonashaag]

Terribly confused about this change. Doesn't 10 * heartbeat_interval() always evaluate to true since heartbeat_interval is strictly positive?

[quasiben]

No, you're right. i messed up the logic here.

[garanews]

Is heartbeat_interval something in the future can be set in config like other Parameters?

[mrocklin]

@garanews this PR has been closed. I encourage you to raise a new issue at https://github.com/dask/distributed/issues/new