vLLM's default multiprocessing method is incompatible with ROCm and Gaudi

[tiran]

Describe the bug
vLLM defaults to VLLM_WORKER_MULTIPROC_METHOD=fork, https://docs.vllm.ai/en/v0.6.1/serving/env_vars.html . Forking is incompatible with ROCm and Gaudi.

To Reproduce

1. Configure InstructLab to use more than one GPU
2. Run ilab model serve on a system with more than one AMD GPU

Expected behavior
vLLM works

Screenshots

RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Additional context
I recommend to switch to "spawn". Python is switching from fork to spawn for all platforms. The fork method has issues, e.g. it can lead to deadlocks when a process mixes threads and fork.

I switch InstructLab to spawn a long time ago, because it was causing trouble on Gaudi, see #956. InstructLab should set VLLM_WORKER_MULTIPROC_METHOD=spawn by default.

[ktam3]

Additional comment by Russell

Indeed, setting the environment variable is the right thing to do in the short term, as it will work with the currently shipped version of vLLM.

FYI, for the future, spawn will be used automatically when you run vllm serve as of v0.6.3. vllm-project/vllm#8823

[nathan-weinberg]

@n1hility given @russellb's comment above, will this ticket be covered by the planned VLLM bump?

[n1hility]

IMO we should fix this in the ODH branches for the vllm versions we are pulling in, and also in any container definitions. I don't have a problem with adding some code as a rededundancy to instructlab to handle plugging in different versions of vllm .

[nathan-weinberg]

@n1hility does ODH vLLM 0.6.2 have this fix or do we need to wait for the next bump?

[n1hility]

Looks like we need to wait for another bump. The branches were created in odh for intel and amd, but not utilized yet and we still need a patch here.

[nathan-weinberg]

@n1hility @tiran do y'all know if ODH vLLM 0.6.4post1 (current vLLM version we are using) or alternatively, 0.6.6post1 (next version we are planning to bump to) has this issue, or could it be closed out?

cc @fabiendupont