-
-
Notifications
You must be signed in to change notification settings - Fork 719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nprocs auto option to dask-worker CLI #4377
Conversation
Kicked CI |
Thanks @quasiben. Looks like it was my fault, new test filename wasn't unique. |
…ker-nproc-auto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just merged the main branch to resolve some unrelated CI failures
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @jacobtomlinson! This is a nice addition. Could you add a test that checks when using --nprocs auto
the expected number of workers with the expected number of threads are created? Something similar to distributed/cli/tests/test_dask_worker.py::test_nprocs_negative
should work here too
Thanks for the feedback @jrbourbeau. Added a test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @jacobtomlinson! This LGTM, will merge after CI finishes
Partly solves #4373
I've moved
nprocesses_nthreads
fromdistributed.deploy.local
intodistributed.deploy.utils
and then am making use of it indistributed.cli.dask_worker
.Setting
nprocs
toauto
when starting a worker will now have the same behaviour asLocalCluster
in terms of the number of processes and threads created.