Does this work deal with the workload balance when scheduing?

[andakai]

No description provided.

[AjayP13]

I assume you're talking about this:
vllm-project/vllm#1237 (comment)

But yes, it does balance the work between multiple instances of the vLLM model.

[andakai]

Yes, I am trying to do dataparallel using vLLM based on your project.
I have read the guide, but still feel confused about how to do this. I wonder if there is any specific doc or example?

[AjayP13]

You can see this test case right here (test_parallel_llm) for some example code:

DataDreamer/src/tests/trainers/test_distributed.py

Line 131 in ad3dd9c

def test_parallel_llm(self, create_datadreamer, mocker):

That test case will run a two copies of an LLM (one per GPU on a 2-GPU machine) using Hugging Face Transformers.

If you want to use VLLM instead of HFTransformers instead, it's as simple as:

from datadreamer.llms import VLLM, ParallelLLM
llm_1 = VLLM("gpt2", device=0)
llm_2 = VLLM("gpt2", device=1)
parallel_llm = ParallelLLM(llm_1, llm_2)

[andakai]

wow this is so easy to use. It helps me a lot. Thanks for your fantastic work.

[AjayP13]

No problem, let me know if you need any other help!