ModuleNotFoundError: No module named 'intel_extension_for_pytorch' when running on aarch64 (EC2 r8g.xlarge)

[kangks]

Hi, I following the steps on https://docs.vllm.ai/en/latest/getting_started/installation/cpu/index.html?device=arm#build-wheel-from-source, but encountered the following error.

How can I avoid building pytorch that requires intel_extension_for_pytorch?

Processed prompts:   0%|                                                       | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/scripts/offline.py", line 412, in <module>
[rank0]:     outputs = llm.chat(messages,
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/entrypoints/llm.py", line 748, in chat
[rank0]:     return self.generate(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/utils.py", line 1057, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/entrypoints/llm.py", line 469, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/entrypoints/llm.py", line 1397, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/engine/llm_engine.py", line 1386, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/executor/executor_base.py", line 284, in execute_model
[rank0]:     driver_outputs = self._driver_execute_model(execute_model_req)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
[rank0]:     return self.driver_worker.execute_model(execute_model_req)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/worker/worker_base.py", line 418, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/worker/cpu_model_runner.py", line 654, in execute_model
[rank0]:     hidden_states = model_executable(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/model_executor/models/qwen2.py", line 486, in forward
[rank0]:     hidden_states = self.model(input_ids, positions, kv_caches,
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/compilation/decorators.py", line 172, in __call__
[rank0]:     return self.forward(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/model_executor/models/qwen2.py", line 348, in forward
[rank0]:     hidden_states, residual = layer(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/model_executor/models/qwen2.py", line 247, in forward
[rank0]:     hidden_states = self.self_attn(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/model_executor/models/qwen2.py", line 179, in forward
[rank0]:     attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/attention/layer.py", line 201, in forward
[rank0]:     return torch.ops.vllm.unified_attention(
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/torch/_ops.py", line 1116, in __call__
[rank0]:     return self._op(*args, **(kwargs or {}))
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/attention/layer.py", line 307, in unified_attention
[rank0]:     return self.impl.forward(self, query, key, value, kv_cache, attn_metadata)
[rank0]:   File "/home/ubuntu/vllm/lib/python3.10/site-packages/vllm-0.7.3.dev178+g69e1d23e.d20250217.cpu-py3.10-linux-aarch64.egg/vllm/attention/backends/torch_sdpa.py", line 537, in forward
[rank0]:     import intel_extension_for_pytorch.llm.modules as ipex_modules
[rank0]: ModuleNotFoundError: No module named 'intel_extension_for_pytorch'

[spaceater]

I followed the steps too,but I didn't encounter the issue while building.Instead,I encountered the same issue when I tried to run a model.

Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]     return self._op(*args, **(kwargs or {}))
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]   File "/root/anaconda3/envs/vLLM/lib/python3.12/site-packages/vllm-0.7.2+cpu-py3.12-linux-x86_64.e>
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]     return self.impl.forward(self, query, key, value, kv_cache, attn_metadata)
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]   File "/root/anaconda3/envs/vLLM/lib/python3.12/site-packages/vllm-0.7.2+cpu-py3.12-linux-x86_64.e>
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139]     import intel_extension_for_pytorch.llm.modules as ipex_modules
Feb 19 10:51:18 kubuntuserver1 bash[4894]: ERROR 02-19 10:51:17 engine.py:139] ModuleNotFoundError: No module named 'intel_extension_for_pytorch'
Feb 19 10:51:26 kubuntuserver1 bash[4874]: ERROR 02-19 10:51:26 client.py:300] RuntimeError('Engine process (pid 4894) died.')
Feb 19 10:51:26 kubuntuserver1 bash[4874]: ERROR 02-19 10:51:26 client.py:300] NoneType: None

It's a ubuntu computer with AMD CPU.

[kangks]

Hi @spaceater thanks for validating.

Yes my bad, the issue was when running my Python script.

Did you enable the prefix caching enable_prefix_caching=True?

I realized the error goes away if I remove that line, which works for me for now, but would like to cache the prompt especially for batch processing to improve the throughput.

[spaceater]

I first used pip to install intel_extension_for_python,but it couldn't work,so I just rebuilt the whole environment.Then I figured out that only 2.5.0 version of intel_extension_for_python was usable in current environment.And then problem solved.
Actually,I don't know why it works,because my CPU is not even Intel CPU.
Anyway,this is the command that I use to start vllm service.

VLLM_LOGGING_LEVEL=WARNING VLLM_CPU_OMP_THREADS_BIND=0-7 vllm serve /root/vLLM/models/DeepSeek-R1-Distill-Qwen-7B --host 127.0.0.1 --port 2998 --max-model-len 2048 --max-num-seqs 1024 --enable-prefix-caching --trust-remote-code --served-model-name DeepSeek-R1-7B