[Performance]: V1 higher memory usage

[wedobetter]

Proposal to improve performance

No response

Report of performance regression

Hardware: 4x RTX 3070 = 32GB VRAM

Issue: I was able to run Qwen/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4 with 12K context length with 0.6.x, now with 0.7.0 + VLLM_USE_V1=1 I cannot push the context length higher than 3K or encountering a CUDA OOM error.
Of course, I can reconfigure it to avoid OOM, my question is: Is V1 expected to consume more memory?

Some of the libraries:

flashinfer==0.1.6+cu124torch2.4
torch==2.5.1
transformers==4.48.1
vllm==0.7.0

VLLM command

        - vllm
        - serve
        - Qwen/Qwen2.5-Coder-32B-Instruct-GPTQ-Int4
        - --gpu-memory-utilization=1
        - --tensor-parallel-size=4
        - --load-format=auto
        - --enforce-eager
        - --swap-space=0
        - --max-model-len=12K
        - --max-num-batched-tokens=12K
        - --disable-fastapi-docs
        - --trust-remote-code
        - --enable-auto-tool-choice
        - --tool-call-parser=hermes

Thanks

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[WoosukKwon]

Hi @wedobetter thanks for reporting the issue. We will take a look at the memory profiling part of V1. Meanwhile, could you please try lower pu-memory-utilization like 0.95 or 0.9? Generally, We don't recommend 1.0 because our memory profiling is not 100% accurate (since we do dynamic memory allocation during run time).

[mgoin]

I think you could also try reducing --max-num-batched-tokens=12K, this could be much smaller like 2k and might reduce peak activations for memory and from torch.compile compilation

[DarkLight1337]

I have noticed similar issues for VLMs as well. See Slack thread. cc @ywang96

[wedobetter]

I think you could also try reducing --max-num-batched-tokens=12K, this could be much smaller like 2k and might reduce peak activations for memory and from torch.compile compilation

As I stated, I know how to reconfigure the execution parameters, my observation was OOM encountered just by upgrading from latest 0.6.x to 0.7.0 VLLM and enabling V1, while keeping all the runtime parameters the same

[wedobetter]

Hi @wedobetter thanks for reporting the issue. We will take a look at the memory profiling part of V1. Meanwhile, could you please try lower pu-memory-utilization like 0.95 or 0.9? Generally, We don't recommend 1.0 because our memory profiling is not 100% accurate (since we do dynamic memory allocation during run time).

Thanks for your input. I generally set that to 1, fearing that CPU offloading can significantly affect performance but I have probably confused gpu-memory-utilization with cpu_offload_gb options.

[robertgshaw2-redhat]

Thanks for the feedback. Note for contributors -- another place to look is torch.compile and the number of cudagraphs we use

[Leon-Sander]

(since we do dynamic memory allocation during run time).

Can you elaborate a bit more on what exactly is dynamic?

[focuzz8]

I can confirm increased memory consumption with Qwen/Qwen2.5-32B-Instruct-AWQ
Hardware: 2x RTX 4070 Ti = 32GB VRAM
I suspect that @robertgshaw2-redhat is correct and problem is related to the torch.compile.

[chelseaztq]

I have the same issue. When deploying the DeepSeek R1 32B distilled model, the GPU memory usage under the v1 engine is higher than under v0. This causes the v1 engine to run out of memory (OOM) under the same configuration, while v0 does not.

[rohan-uiuc]

I have the same issue, blocking me to use V1 arch. I tried using the same configuration as with V0, also tried reducing gpu memory utilization from 0.95 to 0.8 but to no luck always resulting in an OOM on startup.

Also, it takes extremely long for the server to warmup as compared to V0.