Shuffle Service #5976
Shuffle Service #5976
Commits on Mar 15, 2022
-
Handle buffers manually in multi_file
This manages memory more smoothly. We still have issues though in that we're still passing around slices of arrow tables, which hold onto large references
-
This helps to reduce lots of extra unmanaged memory This flow pretty well right now. I'm finding that it's useful to blend between the disk and comm buffer sizes. The abstraction in multi_file and multi_comm are getting a little bit worn down (it would be awkward to shift back to pandas), but maybe that's ok.
Commits on Mar 16, 2022
-
-
-
Change configuration for smoother single-machine use
We don't need a lot of comm buffer, we also don't want more connecitons than machines (too much sitting in buffers). We also improve some printing
-
-
-
Commits on Mar 17, 2022
-
Allow worker extensions to piggy-back on heartbeat
To enable better diagnostics, it would be useful to allow worker extensions to piggy-back on the standard heartbeat. This adds an optional "heartbeat" method to extensions, and, if present, calls a custom method that gets sent to the scheduler and processed by an extension of the same name. This also starts to store the extensions on the worker in a named dictionary. Previously this was a list, but I'm not sure that it was actually used anywhere. This is a breaking change without deprecation, but in a space that I suspect no one will care about. I'm happy to provide a fallback if desired.
-
-
-
-
-
-
-
Commits on Mar 18, 2022
-
-
Add back in manual addition of stealing extension
Tests are failing. I can't reproduce locally. This is just blind praying that it fixes the problem. It should be innocuous.
-
-
-
-
-
-
-
-
Commits on Mar 19, 2022
-
-
Merge branch 'heartbeat-extensions' of github.com:mrocklin/distribute…
…d into heartbeat-extensions
-
-
-
-
-
-
Commits on Mar 22, 2022
-
-
-
-
-
-
-
-
-
-
Update distributed/stealing.py
Co-authored-by: Florian Jetter <fjetter@users.noreply.github.com>
-
-
Merge branch 'heartbeat-extensions' of github.com:mrocklin/distribute…
…d into heartbeat-extensions
Commits on Mar 23, 2022
-
Update distributed/shuffle/multi_file.py
Co-authored-by: Ashwin Srinath <3190405+shwina@users.noreply.github.com>
Commits on Mar 25, 2022
Commits on Mar 28, 2022
-
Move arrow utilities over to a separate file
Also add pyarrow to precommit / mypy settings
Commits on Mar 29, 2022
Commits on Mar 31, 2022
-
-
-
-
-
Cluster Dump SchedulerPlugin (dask#5983)
Add SchedulerPlugin to dump state on cluster close This also adds a new method to SchedulerPlugins that runs directly before closing time
-
-
Update gpuCI
RAPIDS_VERto22.06(dask#5962)Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
-
-
-
Make test_reconnect async (dask#6000)
This was flakey due to cleaning up resources. My experience is that making things async helps with this in general. I don't have strong confidence that this will fix the issue, but I do have mild confidence, and strong confidence that it won't hurt.
-
-
-
-
-
-
-
