-
-
Notifications
You must be signed in to change notification settings - Fork 718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shuffle Service #5976
Shuffle Service #5976
Commits on Mar 13, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 9deb8c6 - Browse repository at this point
Copy the full SHA 9deb8c6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f62911 - Browse repository at this point
Copy the full SHA 3f62911View commit details -
Configuration menu - View commit details
-
Copy full SHA for 626eda0 - Browse repository at this point
Copy the full SHA 626eda0View commit details
Commits on Mar 14, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 7f48f77 - Browse repository at this point
Copy the full SHA 7f48f77View commit details -
Configuration menu - View commit details
-
Copy full SHA for 99bb283 - Browse repository at this point
Copy the full SHA 99bb283View commit details
Commits on Mar 15, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 0d49ab9 - Browse repository at this point
Copy the full SHA 0d49ab9View commit details -
Handle buffers manually in multi_file
This manages memory more smoothly. We still have issues though in that we're still passing around slices of arrow tables, which hold onto large references
Configuration menu - View commit details
-
Copy full SHA for 1e1311d - Browse repository at this point
Copy the full SHA 1e1311dView commit details -
This helps to reduce lots of extra unmanaged memory This flow pretty well right now. I'm finding that it's useful to blend between the disk and comm buffer sizes. The abstraction in multi_file and multi_comm are getting a little bit worn down (it would be awkward to shift back to pandas), but maybe that's ok.
Configuration menu - View commit details
-
Copy full SHA for b99329a - Browse repository at this point
Copy the full SHA b99329aView commit details
Commits on Mar 16, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 5781b7e - Browse repository at this point
Copy the full SHA 5781b7eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ce6e01 - Browse repository at this point
Copy the full SHA 0ce6e01View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3204997 - Browse repository at this point
Copy the full SHA 3204997View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b11d6d - Browse repository at this point
Copy the full SHA 8b11d6dView commit details -
Change configuration for smoother single-machine use
We don't need a lot of comm buffer, we also don't want more connecitons than machines (too much sitting in buffers). We also improve some printing
Configuration menu - View commit details
-
Copy full SHA for 62cc43d - Browse repository at this point
Copy the full SHA 62cc43dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a64248 - Browse repository at this point
Copy the full SHA 5a64248View commit details -
Configuration menu - View commit details
-
Copy full SHA for b901613 - Browse repository at this point
Copy the full SHA b901613View commit details -
Configuration menu - View commit details
-
Copy full SHA for 24092bb - Browse repository at this point
Copy the full SHA 24092bbView commit details
Commits on Mar 17, 2022
-
Allow worker extensions to piggy-back on heartbeat
To enable better diagnostics, it would be useful to allow worker extensions to piggy-back on the standard heartbeat. This adds an optional "heartbeat" method to extensions, and, if present, calls a custom method that gets sent to the scheduler and processed by an extension of the same name. This also starts to store the extensions on the worker in a named dictionary. Previously this was a list, but I'm not sure that it was actually used anywhere. This is a breaking change without deprecation, but in a space that I suspect no one will care about. I'm happy to provide a fallback if desired.
Configuration menu - View commit details
-
Copy full SHA for 7fbe4aa - Browse repository at this point
Copy the full SHA 7fbe4aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 89f6347 - Browse repository at this point
Copy the full SHA 89f6347View commit details -
Configuration menu - View commit details
-
Copy full SHA for bc81db8 - Browse repository at this point
Copy the full SHA bc81db8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 27d2ab3 - Browse repository at this point
Copy the full SHA 27d2ab3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9fd6da0 - Browse repository at this point
Copy the full SHA 9fd6da0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8641abb - Browse repository at this point
Copy the full SHA 8641abbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 34617db - Browse repository at this point
Copy the full SHA 34617dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for e1c0a4d - Browse repository at this point
Copy the full SHA e1c0a4dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c28b83 - Browse repository at this point
Copy the full SHA 8c28b83View commit details
Commits on Mar 18, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 29253b4 - Browse repository at this point
Copy the full SHA 29253b4View commit details -
Add back in manual addition of stealing extension
Tests are failing. I can't reproduce locally. This is just blind praying that it fixes the problem. It should be innocuous.
Configuration menu - View commit details
-
Copy full SHA for 1f79575 - Browse repository at this point
Copy the full SHA 1f79575View commit details -
Configuration menu - View commit details
-
Copy full SHA for cf9a939 - Browse repository at this point
Copy the full SHA cf9a939View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f2286e - Browse repository at this point
Copy the full SHA 6f2286eView commit details -
Configuration menu - View commit details
-
Copy full SHA for efedc04 - Browse repository at this point
Copy the full SHA efedc04View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9b7b03b - Browse repository at this point
Copy the full SHA 9b7b03bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 456de23 - Browse repository at this point
Copy the full SHA 456de23View commit details -
Configuration menu - View commit details
-
Copy full SHA for aef2f61 - Browse repository at this point
Copy the full SHA aef2f61View commit details -
Configuration menu - View commit details
-
Copy full SHA for f79e923 - Browse repository at this point
Copy the full SHA f79e923View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e0256f - Browse repository at this point
Copy the full SHA 1e0256fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 97fb09c - Browse repository at this point
Copy the full SHA 97fb09cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 76baf4b - Browse repository at this point
Copy the full SHA 76baf4bView commit details
Commits on Mar 19, 2022
-
Configuration menu - View commit details
-
Copy full SHA for f58b2e9 - Browse repository at this point
Copy the full SHA f58b2e9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9bc6ce6 - Browse repository at this point
Copy the full SHA 9bc6ce6View commit details -
Merge branch 'heartbeat-extensions' of github.com:mrocklin/distribute…
…d into heartbeat-extensions
Configuration menu - View commit details
-
Copy full SHA for 57b4a42 - Browse repository at this point
Copy the full SHA 57b4a42View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1309c22 - Browse repository at this point
Copy the full SHA 1309c22View commit details -
Configuration menu - View commit details
-
Copy full SHA for c894f40 - Browse repository at this point
Copy the full SHA c894f40View commit details -
Configuration menu - View commit details
-
Copy full SHA for 486320d - Browse repository at this point
Copy the full SHA 486320dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6419328 - Browse repository at this point
Copy the full SHA 6419328View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e6aadc - Browse repository at this point
Copy the full SHA 9e6aadcView commit details -
Configuration menu - View commit details
-
Copy full SHA for c202385 - Browse repository at this point
Copy the full SHA c202385View commit details -
Configuration menu - View commit details
-
Copy full SHA for cf51784 - Browse repository at this point
Copy the full SHA cf51784View commit details -
Configuration menu - View commit details
-
Copy full SHA for c272458 - Browse repository at this point
Copy the full SHA c272458View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d114c2 - Browse repository at this point
Copy the full SHA 1d114c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ba9923 - Browse repository at this point
Copy the full SHA 4ba9923View commit details
Commits on Mar 21, 2022
-
Configuration menu - View commit details
-
Copy full SHA for e7a5143 - Browse repository at this point
Copy the full SHA e7a5143View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0cd7ae - Browse repository at this point
Copy the full SHA b0cd7aeView commit details
Commits on Mar 22, 2022
-
Configuration menu - View commit details
-
Copy full SHA for bd37f49 - Browse repository at this point
Copy the full SHA bd37f49View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e1af62 - Browse repository at this point
Copy the full SHA 6e1af62View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7776ecb - Browse repository at this point
Copy the full SHA 7776ecbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 107b5a0 - Browse repository at this point
Copy the full SHA 107b5a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for dc8a7a4 - Browse repository at this point
Copy the full SHA dc8a7a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6ef62a0 - Browse repository at this point
Copy the full SHA 6ef62a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8550a13 - Browse repository at this point
Copy the full SHA 8550a13View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2e01aa8 - Browse repository at this point
Copy the full SHA 2e01aa8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4486584 - Browse repository at this point
Copy the full SHA 4486584View commit details -
Update distributed/stealing.py
Co-authored-by: Florian Jetter <fjetter@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 01403b9 - Browse repository at this point
Copy the full SHA 01403b9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 97fdf2a - Browse repository at this point
Copy the full SHA 97fdf2aView commit details -
Merge branch 'heartbeat-extensions' of github.com:mrocklin/distribute…
…d into heartbeat-extensions
Configuration menu - View commit details
-
Copy full SHA for 0bd1f89 - Browse repository at this point
Copy the full SHA 0bd1f89View commit details
Commits on Mar 23, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 182dc83 - Browse repository at this point
Copy the full SHA 182dc83View commit details -
Update distributed/shuffle/multi_file.py
Co-authored-by: Ashwin Srinath <3190405+shwina@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 007ea90 - Browse repository at this point
Copy the full SHA 007ea90View commit details
Commits on Mar 25, 2022
-
Configuration menu - View commit details
-
Copy full SHA for d27d0f3 - Browse repository at this point
Copy the full SHA d27d0f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for b50c61a - Browse repository at this point
Copy the full SHA b50c61aView commit details -
Configuration menu - View commit details
-
Copy full SHA for dfc31fd - Browse repository at this point
Copy the full SHA dfc31fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for ea000a3 - Browse repository at this point
Copy the full SHA ea000a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4598577 - Browse repository at this point
Copy the full SHA 4598577View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2993643 - Browse repository at this point
Copy the full SHA 2993643View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe61116 - Browse repository at this point
Copy the full SHA fe61116View commit details -
Configuration menu - View commit details
-
Copy full SHA for adccb02 - Browse repository at this point
Copy the full SHA adccb02View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3aabef0 - Browse repository at this point
Copy the full SHA 3aabef0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2af4974 - Browse repository at this point
Copy the full SHA 2af4974View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1756fb2 - Browse repository at this point
Copy the full SHA 1756fb2View commit details
Commits on Mar 28, 2022
-
Move arrow utilities over to a separate file
Also add pyarrow to precommit / mypy settings
Configuration menu - View commit details
-
Copy full SHA for 6694c84 - Browse repository at this point
Copy the full SHA 6694c84View commit details -
Configuration menu - View commit details
-
Copy full SHA for 154b21f - Browse repository at this point
Copy the full SHA 154b21fView commit details
Commits on Mar 29, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 36956ef - Browse repository at this point
Copy the full SHA 36956efView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2ab401a - Browse repository at this point
Copy the full SHA 2ab401aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 90673d1 - Browse repository at this point
Copy the full SHA 90673d1View commit details
Commits on Mar 31, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 7d8954a - Browse repository at this point
Copy the full SHA 7d8954aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1796804 - Browse repository at this point
Copy the full SHA 1796804View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8de2793 - Browse repository at this point
Copy the full SHA 8de2793View commit details -
Configuration menu - View commit details
-
Copy full SHA for caa852f - Browse repository at this point
Copy the full SHA caa852fView commit details -
Cluster Dump SchedulerPlugin (dask#5983)
Add SchedulerPlugin to dump state on cluster close This also adds a new method to SchedulerPlugins that runs directly before closing time
Configuration menu - View commit details
-
Copy full SHA for f3fb682 - Browse repository at this point
Copy the full SHA f3fb682View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0a1761d - Browse repository at this point
Copy the full SHA 0a1761dView commit details -
Update gpuCI
RAPIDS_VER
to22.06
(dask#5962)Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 7cdb56f - Browse repository at this point
Copy the full SHA 7cdb56fView commit details -
Configuration menu - View commit details
-
Copy full SHA for d0afbb1 - Browse repository at this point
Copy the full SHA d0afbb1View commit details -
Configuration menu - View commit details
-
Copy full SHA for a74fd38 - Browse repository at this point
Copy the full SHA a74fd38View commit details -
Make test_reconnect async (dask#6000)
This was flakey due to cleaning up resources. My experience is that making things async helps with this in general. I don't have strong confidence that this will fix the issue, but I do have mild confidence, and strong confidence that it won't hurt.
Configuration menu - View commit details
-
Copy full SHA for bde718f - Browse repository at this point
Copy the full SHA bde718fView commit details -
Configuration menu - View commit details
-
Copy full SHA for dd857b8 - Browse repository at this point
Copy the full SHA dd857b8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9efb27c - Browse repository at this point
Copy the full SHA 9efb27cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 69bed31 - Browse repository at this point
Copy the full SHA 69bed31View commit details -
Configuration menu - View commit details
-
Copy full SHA for ec71091 - Browse repository at this point
Copy the full SHA ec71091View commit details -
Configuration menu - View commit details
-
Copy full SHA for 016ed25 - Browse repository at this point
Copy the full SHA 016ed25View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8ee5605 - Browse repository at this point
Copy the full SHA 8ee5605View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c3bfb9 - Browse repository at this point
Copy the full SHA 1c3bfb9View commit details -
Configuration menu - View commit details
-
Copy full SHA for fa235ee - Browse repository at this point
Copy the full SHA fa235eeView commit details
Commits on Apr 8, 2022
-
Configuration menu - View commit details
-
Copy full SHA for df4348f - Browse repository at this point
Copy the full SHA df4348fView commit details