Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CogVLM2-Video 多卡推理支持 #162

Open
zhengxingmao opened this issue Jul 22, 2024 · 3 comments
Open

CogVLM2-Video 多卡推理支持 #162

zhengxingmao opened this issue Jul 22, 2024 · 3 comments
Assignees
@zRzRzRzRzRzRzR

Comments

@zhengxingmao
Copy link

Feature request / 功能建议

CogVLM2-Video 多卡推理支持

Motivation / 动机

CogVLM2-Video 多卡推理支持

Your contribution / 您的贡献

是否可以借鉴图片模型的多卡推理机制实现视频模型的多卡推理？
@huangshiyu13
Copy link
Collaborator

简单点就是每张卡部署一个api，然后通过负载均衡对外提供统一的服务

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR self-assigned this Jul 23, 2024
@zRzRzRzRzRzRzR
Copy link
Member

你是希望在多张显存不够的卡部署吗

@zhengxingmao
Copy link
Author

zhengxingmao commented Jul 24, 2024 •
edited
Loading

你是希望在多张显存不够的卡部署吗

在多卡上加速推理，图片模型提供的多卡推理机制目前测试下来只是显存分担，没有加速效果。
或者能够支持单卡batch推理 ，有什么思路可以供借鉴的吗？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@huangshiyu13 @zhengxingmao @zRzRzRzRzRzRzR