Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Respect task ordering when making worker assignments #4922

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Respect task ordering when making worker assignments #4922

wants to merge 7 commits into from
+101 −13

Commits on Jun 17, 2021

  1. Add occupancy to WorkerState

    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    64f9dfa View commit details
    Browse the repository at this point in the history

  2. Respect task ordering when making worker assignments 

    In situations where tasks have many related tasks, and few dependencies
among them, we try to co-schedule those tasks onto similar workers
according to their dask.ordering.  We do this in hopes that this reduces
the burden of communication on their dependents.
    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    bd2aed5 View commit details
    Browse the repository at this point in the history

  3. Clean up condition, we've already passed if we have dependencies

    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    85acab8 View commit details
    Browse the repository at this point in the history

  4. add keyword parameters and comments

    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    dfa06a1 View commit details
    Browse the repository at this point in the history

  5. Only trigger rootish optimization when we have 2x the tasks 

    It's not worth the effort otherwise
    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    e62ddc6 View commit details
    Browse the repository at this point in the history

  6. Check that the worker still exists 

    Thanks tests!
    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    54caae5 View commit details
    Browse the repository at this point in the history

  7. Avoid neighbor scheduling in one-off cases 

    This gets test_scheduler.py::test_reschedule to pass
    @mrocklin
    mrocklin committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    489a3f7 View commit details
    Browse the repository at this point in the history