How to launch GRPO with vLLM on multi-node slurm?

[pbelevich]

How to write sbatch script to run GRPO with vLLM on multiple nodes? What should be --num_processes? Is GRPOTrainer compatible with multinode training?

[Superskyyy]

GRPOTrainer is compatible with multinode training, but the current vLLM isn't. But it will be in near future.

[pbelevich]

@Superskyyy What do you by compatible? I managed to run GRPOTrainer mulitple nodes, but with a bottleneck of a single vLLM on cuda:7 on the first node and idling cuda:7 on the remaining nodes. But the performance is very low.

[Superskyyy]

@Superskyyy What do you by compatible? I managed to run GRPOTrainer mulitple nodes, but with a bottleneck of a single vLLM on cuda:7 on the first node and idling cuda:7 on the remaining nodes. But the performance is very low.

Exactly, the current vllm deployment method is suboptimal due to the limitations of how accelerate and deepspeed use pdsh to launch nodes, it will as you said waste a card on each node by setting cuda visible devices.