Add the heartbeat_interval variable to scheduler Parameters

[garanews]

With the #3896 it was tuned the heartbeat_interval and correlated to number of workers:

I think it would be useful have the heartbeat_interval as a Parameter.
With few workers the heartbeat messages are flooding everything and is hard to debug code:

[mrocklin]

Correct, right now heartbeat intervals are adaptive based on the number of workers, and there is no option to make this configurable.

In principle having a configuration value that defaulted to "auto" which would imply the current behavior seems sensible to me.

Short term, you could also change your logging level. I tend to use INFO by default.

[garanews]

Thanks @mrocklin , in case I would set logging INFO I am reading here https://docs.dask.org/en/latest/configuration.html and see that is possible to set with YAML, with system ENV or inline.
I am wondering if is possible to pass it as command.
For example, this is docker-compose of worker:

  worker01:
    build:
      context: .
      dockerfile: ./compose/local/dask/Dockerfile
      args:
        - http_proxy
        - https_proxy
    image: daskdev/dask
    hostname: dask-worker
    command:
      [
        "dask-worker",
        "--preload",
        "/tmp/daskworkerinit.py",
        "--nthreads",
        "1",
        "--nprocs",
        "1",
        "--memory-limit",
        "3G",
        "tcp://scheduler:8786",
      ]

So I would pass the logging to scheduler:

  scheduler:
    build:
      context: .
      dockerfile: ./compose/local/dask/Dockerfile
      args:
        - http_proxy
        - https_proxy
    image: daskdev/dask
    hostname: dask-scheduler
    ports:
      - "8786:8786"
      - "8787:8787"
    command: ["dask-scheduler"]
    networks:
      - elastic

But not sure if is possible and how :)

[mrocklin]

I'm not that familiar with docker-compose unfortunately. Does it allow you to add files or specify environment variables? If so, then I would use one of those.

…

On Mon, Aug 31, 2020 at 2:46 AM garanews ***@***.***> wrote: Thanks @mrocklin <https://github.com/mrocklin> , in case I would set logging INFO I am reading here https://docs.dask.org/en/latest/configuration.html and see that is possible to set with YAML, with system ENV or inline. I am wondering if is possible to pass it as command. For example, this is docker-compose of worker: worker01: build: context: . dockerfile: ./compose/local/dask/Dockerfile args: - http_proxy - https_proxy image: daskdev/dask hostname: dask-worker command: [ "dask-worker", "--preload", "/tmp/daskworkerinit.py", "--nthreads", "1", "--nprocs", "1", "--memory-limit", "3G", "tcp://scheduler:8786", ] So I would pass the logging to scheduler: scheduler: build: context: . dockerfile: ./compose/local/dask/Dockerfile args: - http_proxy - https_proxy image: daskdev/dask hostname: dask-scheduler ports: - "8786:8786" - "8787:8787" command: ["dask-scheduler"] networks: - elastic But not sure if is possible and how :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4083 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTDNMZG4DHCRPE6AMSDSDNWRFANCNFSM4QOGAP7A> .

[garanews]

For sure you can pass option to command (as I already did for dask-worker passing processes, memory..) , but I see that dask-scheduler doesn't have logging level as option:

[quasiben]

I would probably start with passing in the the logging level as an env var with docker compose. That would be something like:

worker01:
  environment:
    - DASK_DISTRIBUTED__LOGGING__DISTRIBUTED="info"

Note: I used our new handy conversion tool to help with this: https://docs.dask.org/en/latest/configuration.html#conversion-utility

Thanks @jacobtomlinson

[garanews]

thanks, I passed that and I see it is set inside the container, but still logging the debug messages...
so if I enter in scheduler container and check env with command export I have:
export DASK_DISTRIBUTED__LOGGING__DISTRIBUTED='info'
export DASK_DISTRIBUTED__LOGGING__WORKER='info'
To be sure I set also these 2 in worker container but still logging the debug messages.

So question is how to avoid these messages? The 2 variables I set previously don't seem to work:

worker01_1 | DEBUG 2020-09-03 09:54:16,758 worker 363 140492157343552 Heartbeat: tcp://172.18.0.7:43423
worker01_1 | DEBUG 2020-09-03 09:54:17,200 worker 365 140077030909760 Heartbeat: tcp://172.18.0.7:36285
worker01_1 | DEBUG 2020-09-03 09:54:17,258 worker 363 140492157343552 Heartbeat: tcp://172.18.0.7:43423

[quasiben]

Are you setting the logging level to debug elsewhere in your setup ? Typically debug is something the user has to configure and the default is info

[garanews]

ehm, I just using docker image daskdev/dask so nothing touch about debug logging.
I see image is called daskdev so understandable debug level, but unfortunately there is not an image daskprod so....

[quasiben]

Hmm, I just tried the latest from https://github.com/dask/dask-docker and I only have INFO output. Is there some in your ~/.config/dask/distributed.yaml file which has DEBUG on ? Or something which changes logging behavior in daskworkerinit.py ?

[garanews]

I found it! I had DEBUG level in base.py :
"loggers": {"distributed": {"level": "DEBUG", "handlers": ["console"]},},

So now finally I do not have hearbeat log anymore! Thanks @quasiben for your time!

[quasiben]

Thanks @garanews -- closing