Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP co-assign related root-ish tasks #4899

Closed

Commits on Jun 15, 2021

  1. Configuration menu
    Copy the full SHA
    7f454fd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8e8f7f1 View commit details
    Browse the repository at this point in the history
  3. matt's occupancy-based method

    Unfortunately `ts._prefix._duration_average` == -1 for all the root tasks we care about, so this won't work.
    gjoseph92 committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    b12d490 View commit details
    Browse the repository at this point in the history
  4. Revert "matt's occupancy-based method"

    This reverts commit e348a7c.
    gjoseph92 committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    064be2e View commit details
    Browse the repository at this point in the history
  5. don't require dependents

    gjoseph92 committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    1484e65 View commit details
    Browse the repository at this point in the history
  6. pass in total_nthreads

    gjoseph92 committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    5163e77 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d9df8be View commit details
    Browse the repository at this point in the history
  8. REVERTME print statements

    gjoseph92 committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    7b9728f View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2021

  1. Ignore deps of root-likes. This is working well.

    When a task is root-like and the previous worker is full, we don't want to use the normal `decide_worker` logic, since that only considers as candidates workers that have the deps of the dask. Since the task only has 1-5 deps, we'd only ever consider the same 1-5 workers.
    gjoseph92 committed Jun 16, 2021
    Configuration menu
    Copy the full SHA
    0fbb75e View commit details
    Browse the repository at this point in the history
  2. Save a few cycles

    Only compute `total_nthreads` when a new worker is needed, and only compute the number of dependencies once per task group. Overloads the meaning of `_last_worker` to indicate if we've decided in the past whether a TaskGroup is root-ish or not.
    gjoseph92 committed Jun 16, 2021
    Configuration menu
    Copy the full SHA
    f2da0bc View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2021

  1. WIP docs

    gjoseph92 committed Jun 17, 2021
    Configuration menu
    Copy the full SHA
    d3db281 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c929c96 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2021

  1. Revert "Save a few cycles"

    This reverts commit f2da0bc.
    gjoseph92 committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    f25ed42 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2021

  1. Configuration menu
    Copy the full SHA
    f50daf1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3a73508 View commit details
    Browse the repository at this point in the history
  3. Tests

    TODO test out of priority order
    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    cfe37f6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0b5486f View commit details
    Browse the repository at this point in the history
  5. Revert "Test both axes. I think this is excessive."

    This reverts commit 0d1e2380b525b0ee7b4e60d9bee62f889ed5520b.
    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    2c2bb68 View commit details
    Browse the repository at this point in the history
  6. Handle zero-division

    Maybe avoiding -1 is excessive; I just wanted it to still work if we changed to a Py_ssize_t
    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    5e58b5a View commit details
    Browse the repository at this point in the history
  7. Unset _last_worker when we lose a worker

    Arguably would be better to check that `ws` is in (valid_workers or all_workers); downside is that the common `all_workers` would require an O(n) search, since it's only `dict_values`, not `dict`.
    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    4132583 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    df2cf70 View commit details
    Browse the repository at this point in the history
  9. comment out print

    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    19107d8 View commit details
    Browse the repository at this point in the history
  10. Remove some cruft

    gjoseph92 committed Jun 23, 2021
    Configuration menu
    Copy the full SHA
    8e45244 View commit details
    Browse the repository at this point in the history