Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Implement accurate memory accounting for all-to-all operations #50290

Merged
merged 42 commits into from
Feb 15, 2025
Merged

[Data] Implement accurate memory accounting for all-to-all operations #50290

merged 42 commits into from
Feb 15, 2025

Conversation

bveeramani
Copy link
Member

@bveeramani bveeramani commented Feb 6, 2025 •
edited
Loading

Why are these changes needed?

AllToAllOperator and ZipOperator don't implement accurate memory. As a result, if a plan contains either of these operators, the streaming executors falls back to the legacy scheduling algorithm.

Related issue number

Fixes #48104

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@bveeramani
Initial commit
cb830b0 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
1d9dd2e 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
bed2558 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
c40b06d 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
49a8d71 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
cad41d5 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
17df280 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
bb0e778 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
2620239 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
4dff29a 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
0393f97 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
2a92d8e 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
7d912a8 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
e38bf85 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
01c8911 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
a7e6cb2 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
53dfa51 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
27b1315 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
ba39108 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
6f9d698 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
11c498f 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
ed5fd50 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
1209f80 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
f992321 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
59a732b 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani bveeramani marked this pull request as draft February 6, 2025 18:40
@bveeramani
Fix bug
adcf0b6 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Remove old method
a5feb62 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
bveeramani and others added 13 commits February 10, 2025 15:05
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray
29e12e4 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Address review comments
ba1d228 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Initial commit
a84080d 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'union-fix' into legacy-schedule
d61c3e0 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Fix bug
e1f129c 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Fxi bug
b11ffe5 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Simplify test
22868d0 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Test doctest
3b16e38 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Fix ordering bug
de74da5 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Fix ordering bug
d597f1d 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'union-fix' into legacy-schedule
a5c23e5 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Merge branch 'master' into legacy-schedule
d3ecd31
@bveeramani
Uppdate files
141af8b 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani bveeramani added the go add ONLY when ready to merge, run all tests label Feb 14, 2025
@bveeramani
Fix bug
70c2521 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Fix tests
6287ebd 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani bveeramani marked this pull request as ready for review February 14, 2025 23:51
@bveeramani bveeramani requested a review from a team as a code owner February 14, 2025 23:51
bveeramani
bveeramani commented Feb 14, 2025
View reviewed changes
python/ray/data/_internal/execution/operators/base_physical_operator.py
Comment on lines +167 to +168
def implements_accurate_memory_accounting(self):
return True
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove this method altogether in a follow-up PR. Wanted to keep this PR relatively small

@bveeramani bveeramani enabled auto-merge (squash) February 15, 2025 00:30
@bveeramani bveeramani merged commit f0942dd into master Feb 15, 2025
6 checks passed
@bveeramani bveeramani deleted the legacy-schedule branch February 15, 2025 00:38
400Ping pushed a commit to 400Ping/ray that referenced this pull request Feb 20, 2025
@bveeramani @400Ping
[Data] Implement accurate memory accounting for all-to-all operations (…
ca66e82 
…ray-project#50290)

`AllToAllOperator` and `ZipOperator` don't implement accurate memory. As
a result, if a plan contains either of these operators, the streaming
executors falls back to the legacy scheduling algorithm.

Fixes ray-project#48104

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com>
israbbani pushed a commit that referenced this pull request Feb 25, 2025
@bveeramani @israbbani
[Data] Implement accurate memory accounting for all-to-all operations (…
330723c 
…#50290)

`AllToAllOperator` and `ZipOperator` don't implement accurate memory. As
a result, if a plan contains either of these operators, the streaming
executors falls back to the legacy scheduling algorithm.

Fixes #48104

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
xsuler pushed a commit to antgroup/ant-ray that referenced this pull request Mar 4, 2025
@bveeramani @xsuler
[Data] Implement accurate memory accounting for all-to-all operations (…
86ad24a 
…ray-project#50290)

`AllToAllOperator` and `ZipOperator` don't implement accurate memory. As
a result, if a plan contains either of these operators, the streaming
executors falls back to the legacy scheduling algorithm.

Fixes ray-project#48104

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
xsuler pushed a commit to antgroup/ant-ray that referenced this pull request Mar 4, 2025
@bveeramani @xsuler
[Data] Implement accurate memory accounting for all-to-all operations (…
f9b4561 
…ray-project#50290)

`AllToAllOperator` and `ZipOperator` don't implement accurate memory. As
a result, if a plan contains either of these operators, the streaming
executors falls back to the legacy scheduling algorithm.

Fixes ray-project#48104

---------

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexeykudinkin alexeykudinkin alexeykudinkin left review comments

@raulchen raulchen raulchen approved these changes

Assignees
No one assigned
Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Data] Ray Data runs out of disk while writing Parquet files
3 participants
@bveeramani @alexeykudinkin @raulchen