Use task-based rechunking to prechunk along partial boundaries #8831
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 25 files ± 0 25 suites ±0 10h 20m 20s ⏱️ + 8m 25s For more details on these failures, see this check. Results for commit 245f345. ± Comparison against base commit fe79a36. ♻️ This comment has been updated with latest results. |
hendrikmakait
force-pushed
the
use-rechunking-to-prechunk
branch
from
d1a204c to
c23fd8f
Compare
.pre-commit-config.yaml
Outdated
| @@ -64,7 +64,7 @@ repos: | |||
| - tornado | |||
| - pyarrow | |||
| - urllib3 | |||
| - git+https://github.com/dask/dask | |||
| - git+https://github.com/hendrikmakait/dask@prechunk-p2p-rechunk | |||
There was a problem hiding this comment.
TODO: Revert before merging
distributed/shuffle/_rechunk.py
Outdated
| @@ -431,52 +432,108 @@ def _construct_graph(self) -> _T_LowLevelGraph: | |||
| return dsk | |||
|
|
|||
|
|
|||
| def _prechunk_for_partials( | |||
There was a problem hiding this comment.
The function name implies to me that we apply the rechunking here, could you rename to something that tells us that we are only calculating the chunks?
| [((2, 2),), ((2, 2),), ((2, 2),)], | ||
| [((2, 2),), ((4,),), ((2, 2),)], | ||
| [((2, 2),), ((1, 1, 1, 1),), ((2, 2),)], | ||
| [((2, 2, 2),), ((1, 2, 2, 1),), ((1, 1, 1, 1, 2),)], |
There was a problem hiding this comment.
Why is the last 2 in expected a good idea?
There was a problem hiding this comment.
Is it because the 1 in new at the end means that the input chunk only creates 2 output chunks?
There was a problem hiding this comment.
I've thought about this a little more, and you're right! There was a small bug here. I've adjusted the algorithm accordingly.
|
looks like there is an env mismatch of some kind because of tags |
This PR faciliates P2P rechunking by offloading pre-chunking along partial boundaries to task-based rechunking. This also functions as a predecessor to pre-concatenation of small chunks which can also be offloaded to P2P rechunking.
pre-commit run --all-files