[Core][Distributed] add cleanup code for model parallel

[youkaichao]

I did a quick search and find that only test code calls destroy_model_parallel:

vllm/tests/lora/conftest.py

Line 26 in d627a3d

destroy_model_parallel()

vllm/tests/conftest.py

Line 50 in d627a3d

destroy_model_parallel()

This means user's code never called destroy_model_parallel.

[zhuohan123]

Why is this needed?

[youkaichao]

I met some strange cleanup bugs today, and find the bug is in the cleanup logic.

I think the problem is, user's code never called destroy_model_parallel , so the resource of model parallel group is never released, leading to strange errors like #3829 (comment) (which is "fixed" in #3702 ).

However, I admit that I'm not confident with this. The bug is unstable and difficult to reproduce.

[simon-mo]

but this is called?

vllm/vllm/distributed/device_communicators/pynccl.py

Lines 282 to 285 in fa32207

	def __del__(self):
	# `dist` module might have been already destroyed
	if hasattr(dist, 'destroy_process_group'):
	dist.destroy_process_group()

[youkaichao]

close in favor of #4508 .