[Bug] baichuan-13b-chat Service exception after long run

[Tomorrowxxy]

Start command

python -m vllm.entrypoints.openai.api_server --model baichuan-inc/Baichuan-13B-Chat --host 0.0.0.0 --port 8777 --trust-remote-code --dtype half

After about 12 hours of operation, the inference service stopped working

GPU：V100
CUDA：11.4

Screenshot of the problem：

[zhuohan123]

Can you describe in more detail what exactly happened? For example, does all future requests fail, or just one specific request fail?

From the screenshot it feels like it's maybe because the client disconnects and thus the server stops the running request.

[Tomorrowxxy]

@zhuohan123

• All future requests will fail, there is no Avg prompt throughput: xxxx reasoning and other related information after Received request, directly Aborted request
• That is, the current machine is not processing any inference requests and must manually kill the process and restart the service
• This happened after running for about 6 hours
  
  

[Tomorrowxxy]

• All future requests will fail, there is no Avg prompt throughput: xxxx reasoning and other related information after Received request, directly Aborted request
• That is, the current machine is not processing any inference requests and must manually kill the process and restart the service
• This happened after running for about 6 hours
  
  

I think it should be caused by insufficient CUDA memory. As shown in the picture, the occupancy has reached 95%, resulting in no more reasoning
I tried to start baichuan-13b with gpu_memory_utilization = 0.8 on v100, sorry for the failure to start, it must be 0.9.

[Tomorrowxxy]

After running for a while, no more inference. vllm's service is still there
Often appear, how should it be solved? @zhuohan123

[xiaocode337317439]

+1