Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ray] fix local_rank issue #2596

Merged
merged 3 commits into from
Jan 19, 2021
Merged

[ray] fix local_rank issue #2596

merged 3 commits into from
Jan 19, 2021

Conversation

richardliaw
Copy link
Collaborator

@richardliaw richardliaw commented Jan 14, 2021 •
edited by yuduber
Loading

Checklist before submitting

  • Did you read the contributor guide?
  • Did you update the docs?
  • Did you write any tests to validate this change?
  • Did you update the CHANGELOG, if this change affects users?

Description

Fixes #2583

Review process to land

  1. All tests and other checks must succeed.
  2. At least one member of the technical steering committee must review and approve.
  3. If any member of the technical steering committee requests changes, they must be addressed.

@richardliaw
update
ed2bd89 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@richardliaw
changelog
787350c 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@richardliaw
Copy link
Collaborator Author

cc @tgaddair

@richardliaw richardliaw mentioned this pull request Jan 14, 2021
Closed
@github-actions

This comment has been minimized.

Sign in to view
tgaddair
tgaddair reviewed Jan 15, 2021
View reviewed changes
horovod/ray/runner.py Outdated
os.environ["HOROVOD_HOSTNAME"] = self.hostname()
os.environ["HOROVOD_RANK"] = str(world_rank)
os.environ["HOROVOD_LOCAL_RANK"] = str(local_rank)
os.environ["HOROVOD_LOCAL_SIZE"] = str(local_size)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may also need to set cross rank and cross size.

@richardliaw
fix-tests
d6d2241 
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
@github-actions
Copy link

Unit Test Results

     552 files   -   32     552 suites   - 32   4h 49m 31s ⏱️ - 4m 28s
     538 tests ±    0     509 ✔️ +    1       29 💤 ±    0  0 ❌  - 1 
11 286 runs   - 604  8 957 ✔️  - 489  2 329 💤  - 114  0 ❌  - 1 

Results for commit d6d2241. ± Comparison against base commit a9dea74.

@richardliaw
Copy link
Collaborator Author

should be ready for another review!

tgaddair
tgaddair approved these changes Jan 19, 2021
View reviewed changes
Copy link
Collaborator

@tgaddair tgaddair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tgaddair tgaddair merged commit 96dd0ad into horovod:master Jan 19, 2021
irasit pushed a commit to irasit/horovod that referenced this pull request Jan 29, 2021
@richardliaw @irasit
[ray] fix local_rank issue (horovod#2596)
86dbd19
irasit pushed a commit to irasit/horovod that referenced this pull request Jan 29, 2021
@richardliaw @irasit
[ray] fix local_rank issue (horovod#2596)
abfb7e3 
Signed-off-by: Peng Zhang <pengz@uber.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tgaddair tgaddair tgaddair approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Horovod on Ray has all local_rank as 0
2 participants
@richardliaw @tgaddair