Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to select specific gpu index when using tensor parallel? #283

Closed
frankxyy opened this issue Nov 11, 2023 · 2 comments · Fixed by #331
Closed

How to select specific gpu index when using tensor parallel? #283

frankxyy opened this issue Nov 11, 2023 · 2 comments · Fixed by #331
Assignees
@mrwyattii

Comments

@frankxyy
Copy link

frankxyy commented Nov 11, 2023 •
edited
Loading

Hi，I use the command CUDA_VISIBLE_DEVICES=1,3 deepspeed --num_gpus 2 infer.py, but the program is still running on gpu 0 and gpu 1. The same problem also happens when I try to use persistent serving. So how can I arbitrarily select specific gpu index?
@mrwyattii mrwyattii self-assigned this Nov 13, 2023
@mrwyattii
Copy link
Contributor

Hi @frankxyy, this is a feature we have not brought into the latest MII release, but it is available in MII-Legacy via the deploy_rank option in mii_config. I will work to bring this functionality into the latest release. Look for updates soon, thanks!

@yaof20
Copy link

yaof20 commented Nov 26, 2023

Thx for your wonderful work.

I have the same issue. It would be great to see this feature in the latest release!

Please let me know when it comes true :)

@mrwyattii mrwyattii mentioned this issue Nov 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrwyattii mrwyattii

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add device map config deepspeedai/DeepSpeed-MII
3 participants
@frankxyy @mrwyattii @yaof20