[Bug]: When I use cpu VLLM, it show [IndexError: list index out of range]

[WangRongsheng]

Your current environment

/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/model_executor/model_loader/weight_utils.py:424: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state = torch.load(bin_file, map_location="cpu")
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.28it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.27it/s]

INFO 10-03 00:10:12 cpu_executor.py:212] # CPU blocks: 7281
Processed prompts:   0%|                                                                                                    | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/root/autodl-tmp/test_vllm_cpu.py", line 17, in <module>
[rank0]:     outputs = llm.generate(prompts, sampling_params)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/utils.py", line 1051, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/entrypoints/llm.py", line 391, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/entrypoints/llm.py", line 899, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/engine/llm_engine.py", line 1407, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/executor/cpu_executor.py", line 227, in execute_model
[rank0]:     output = self.driver_method_invoker(self.driver_worker,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/executor/cpu_executor.py", line 377, in _driver_method_invoker
[rank0]:     return getattr(driver, method)(*args, **kwargs)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/worker/worker_base.py", line 303, in execute_model
[rank0]:     inputs = self.prepare_input(execute_model_req)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/worker/worker_base.py", line 291, in prepare_input
[rank0]:     return self._get_driver_input_and_broadcast(execute_model_req)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/worker/worker_base.py", line 253, in _get_driver_input_and_broadcast
[rank0]:     self.model_runner.prepare_model_input(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/worker/cpu_model_runner.py", line 498, in prepare_model_input
[rank0]:     sampling_metadata = SamplingMetadata.prepare(seq_group_metadata_list,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/model_executor/sampling_metadata.py", line 160, in prepare
[rank0]:     ) = _prepare_seq_groups(seq_group_metadata_list, seq_lens, query_lens,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm-0.6.3.dev65+g7f60520d.cpu-py3.10-linux-x86_64.egg/vllm/model_executor/sampling_metadata.py", line 287, in _prepare_seq_groups
[rank0]:     query_len = query_lens[i] if query_lens is not None else 1
[rank0]: IndexError: list index out of range
Processed prompts:   0%|   

Model Input Dumps

No response

🐛 Describe the bug

my code is :

import os
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
 
os.environ['VLLM_TARGET_DEVICE'] = 'cpu'
 
model_dir = './AI-ModelScope/Qwen2___5-7B-Instruct'
 
# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_dir,
    local_files_only=True,
)
 
# Prompt
messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': '天空为什么是蓝色的？'}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
llm = LLM(
    model=model_dir,
    device='cpu',
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05)

print(type(text))

outputs = llm.generate([text], sampling_params)
 
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
 
    print(f'Prompt提示词: {prompt!r}, 大模型推理输出: {generated_text!r}')

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Imss27]

This change is recently added by #8839. And it seems like not a complete work yet.
I would suggest to use a stable release version for your usage.

cc: @LiuXiaoxuanPKU for more comments if she is available.

[WangRongsheng]

it can be solved by using a stable release version. thank you!