question: Is it possible to avoid ray in single machine multiple GPUs serving?

[gaocegege]

I'm uncertain whether it's feasible to bypass Ray when serving on a single machine with multiple GPUs. Ray introduces additional maintenance costs in this use case.

[irasin]

At least ray is needed if you want to use tensor parallel with multiple GPUS, since each Worker instance should exist in a single process but not thread. However, we can just replace ray with multiprocess in this regard.

I haven't seen the other reason why we need ray in the code, maybe there are something, for example, memory issue, object sharing or some other stuff.

[hmellor]

Closing because it appears Ray is only used if:

• Specified by the user https://docs.vllm.ai/en/latest/models/engine_args.html#cmdoption-worker-use-ray
• Tensor parallel is used https://docs.vllm.ai/en/latest/models/engine_args.html#cmdoption-worker-use-ray