[V1] Feedback Thread #12568

simon-mo · 2025-01-30T02:46:45Z

Please leave comments here about your usage of V1, does it work? does it not work? which feature do you need in order to adopt it? any bugs?

For bug report, please file it separately and link the issue here.

For in depth discussion, please feel free to join #sig-v1 in the vLLM Slack workspace.

robertgshaw2-redhat · 2025-01-30T02:50:50Z

[Bug]: V1 Regression: ValueError: could not broadcast input array from shape (y,) into shape (x,) #12567

wedobetter · 2025-01-30T07:22:45Z

👍 I have not done a proper benchmark but V1 feels superior, i.e. higher throughput + lower latency, TTFT.
The other thing that I have noticed is that logging has changed Running: 1 reqs, Waiting: 0 reqs, it used to print stats such token/s.

I have encountered a possible higher memory consumption issue, but am overall very pleased with the vllm community's hard work on V1.
#12529

m-harmonic · 2025-01-30T18:30:35Z

Does anyone know about this bug with n>1? Thanks
#12584

robertgshaw2-redhat · 2025-01-30T18:46:50Z

Does anyone know about this bug with n>1? Thanks #12584

Thanks, we are aware and have some ongoing PRs for it.

#10980

robertgshaw2-redhat · 2025-01-30T22:05:54Z

I have encountered a possible higher memory consumption issue, but am overall very pleased with the vllm community's hard work on V1.

Logging is in progress. Current main has a lot more and we will maintain compatibility with V0. Thanks!

dchichkov · 2025-01-30T22:15:04Z

Quick feedback [VLLM_USE_V1=1]:

n > 1 would be nice
guided_grammar (or anything guided really) would be nice

robertgshaw2-redhat · 2025-01-31T02:21:58Z

Quick feedback [VLLM_USE_V1=1]:

n > 1 would be nice

guided_grammar (or anything guided really) would be nice

Thanks, both are in progress

hibukipanim · 2025-01-31T14:29:18Z

are logprobs output (and specifically prompt logprobs with echo=True) expected to be working with current V1 (0.7.0)?
checking here before opening an issue to reproduce

akshay-loci · 2025-01-31T15:16:30Z

Maybe there is a better place to discuss this but the implementation for models that use more than one extra modality is quite non-intuitive. get_multimodal_embeddings() expects that we return a list or tensor of length equal to the number of multimodal items provided in the batch and we then have to make unintuitive assumptions on how the output passed into get_input_embeddings would look like because the batching being used while calling both functions is not the same. It would be much nicer if for example the input and output of get_multimodal_embeddings are dicts with the keys being the different modalities.

robertgshaw2-redhat · 2025-01-31T23:13:12Z

are logprobs output (and specifically prompt logprobs with echo=True) expected to be working with current V1 (0.7.0)? checking here before opening an issue to reproduce

Still in progress

wedobetter · 2025-02-02T14:06:59Z

👍 I have not done a proper benchmark but V1 feels superior, i.e. higher throughput + lower latency, TTFT. The other thing that I have noticed is that logging has changed Running: 1 reqs, Waiting: 0 reqs, it used to print stats such token/s.

I have encountered a possible higher memory consumption issue, but am overall very pleased with the vllm community's hard work on V1. #12529

Thanks for fixing metrics logs in 0.7.1!
Lack of pipeline parallelism in V1 is a show stopper for production deployments #11945

Ouna-the-Dataweaver · 2025-02-03T04:33:35Z

I'm either going insane, but with V1 qwen 8b instruct LLM just breaks in fp8 and around 25% of generations are just gibberish, with same running code and everything. Do I need to make a bug report, or it's an expected behaviour and I need some specific setup of sampling params for it to work in v1?

FrederickVu · 2025-02-03T06:45:20Z

The V1 engine doesn't seem to support logits processors or min-p filtering. Issue #12678

gmonair · 2025-02-03T15:12:28Z

Something is weird with memory calculation in V1 and tensor parallel. Here are 2 cases that I tested recently:

vllm 0.7.0 on 2x A6000:

Starting normally a 32b-awq model and using --max-model-len 32768 --gpu-memory-utilization 0.98 --tensor-parallel 2 --max-num-batched-tokens 32768 --max-seq-len-to-capture 32768

Everything works as previously, GPUs both get to ~44-46GB usage

Using VLLM_USE_V1=1 and the exact same parameters as above:

GPUs both load up to ~24-25GB and it slowly goes up as inference runs. I've seen it go up to 32GB on each GPU.

Updating to vllm 0.7.1 and running a 7b-awq model this time, I also noticed that running the above command "normally" the logs show Maximum concurrency at 44x

Using V1 I get:

INFO 02-02 23:26:19 kv_cache_utils.py:400] Maximum concurrency for 32768 tokens per request: **22.25x**

And finally, with vllm 0.7.0 and 4x L4 loading a 32b-awq model with tp 4 works in "normal mode", but OOMs with V1.

Xarbirus · 2025-02-03T16:33:44Z

I did a little experiment with DeepSeek-R1 on 8xH200 GPU.

vLLM 0.7.0 showed the following results with benchmark_serving.py --backend openai --base-url http://0.0.0.0:8000 --dataset-name=random --model deepseek-ai/DeepSeek-R1

with VLLM_USE_V1=1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [07:53<00:00,  2.11it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  473.62    
Total input tokens:                      1024000   
Total generated tokens:                  119550    
Request throughput (req/s):              2.11      
Output token throughput (tok/s):         252.42    
Total Token throughput (tok/s):          2414.51   
---------------Time to First Token----------------
Mean TTFT (ms):                          100636.33 
Median TTFT (ms):                        103588.53 
P99 TTFT (ms):                           197277.97 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          177.82    
Median TPOT (ms):                        172.14    
P99 TPOT (ms):                           363.05    
---------------Inter-token Latency----------------
Mean ITL (ms):                           173.08    
Median ITL (ms):                         136.46    
P99 ITL (ms):                            575.30    
==================================================

without VLLM_USE_V1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [05:24<00:00,  3.08it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  324.29    
Total input tokens:                      1024000   
Total generated tokens:                  119163    
Request throughput (req/s):              3.08      
Output token throughput (tok/s):         367.46    
Total Token throughput (tok/s):          3525.12   
---------------Time to First Token----------------
Mean TTFT (ms):                          29022.37  
Median TTFT (ms):                        32492.50  
P99 TTFT (ms):                           54457.59  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.16    
Median TPOT (ms):                        119.91    
P99 TPOT (ms):                           411.21    
---------------Inter-token Latency----------------
Mean ITL (ms):                           120.20    
Median ITL (ms):                         76.78     
P99 ITL (ms):                            656.11    
==================================================

In general, vLLM without VLLM_USE_V1 looked more productive. I also tried V0 with --request-rate 10 and got

Traffic request rate: 10.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [05:16<00:00,  3.16it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  316.20    
Total input tokens:                      1024000   
Total generated tokens:                  119448    
Request throughput (req/s):              3.16      
Output token throughput (tok/s):         377.76    
Total Token throughput (tok/s):          3616.21   
---------------Time to First Token----------------
Mean TTFT (ms):                          100122.09 
Median TTFT (ms):                        98699.05  
P99 TTFT (ms):                           201732.11 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          139.61    
Median TPOT (ms):                        104.30    
P99 TPOT (ms):                           1276.91   
---------------Inter-token Latency----------------
Mean ITL (ms):                           105.90    
Median ITL (ms):                         76.35     
P99 ITL (ms):                            648.36    
==================================================

Throughput was still 2 times lower than SGLang in the same benchmark. Today I updated vLLM to the new version (0.7.1) and decided to repeat the experiment. And the results in version V0 have become much better!

without VLLM_USE_V1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [04:29<00:00,  3.71it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  269.74    
Total input tokens:                      1024000   
Total generated tokens:                  119805    
Request throughput (req/s):              3.71      
Output token throughput (tok/s):         444.14    
Total Token throughput (tok/s):          4240.35   
---------------Time to First Token----------------
Mean TTFT (ms):                          368.78    
Median TTFT (ms):                        269.07    
P99 TTFT (ms):                           3826.70   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          124.95    
Median TPOT (ms):                        122.03    
P99 TPOT (ms):                           214.93    
---------------Inter-token Latency----------------
Mean ITL (ms):                           123.32    
Median ITL (ms):                         75.30     
P99 ITL (ms):                            583.77    
==================================================

without VLLM_USE_V1 (with --request-rate 10)

Traffic request rate: 10.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [02:26<00:00,  6.83it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  146.43    
Total input tokens:                      1024000   
Total generated tokens:                  119701    
Request throughput (req/s):              6.83      
Output token throughput (tok/s):         817.48    
Total Token throughput (tok/s):          7810.75   
---------------Time to First Token----------------
Mean TTFT (ms):                          14575.11  
Median TTFT (ms):                        13606.50  
P99 TTFT (ms):                           29954.96  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          297.01    
Median TPOT (ms):                        282.46    
P99 TPOT (ms):                           1393.69   
---------------Inter-token Latency----------------
Mean ITL (ms):                           262.67    
Median ITL (ms):                         132.89    
P99 ITL (ms):                            2840.40   
==================================================

But running vLLM with VLLM_USE_V1=1 I got en error TypeError: FlashAttentionImpl.__init__() got an unexpected keyword argument 'q_lora_rank' with previous warnings like

`torch.compile` is turned on, but the model deepseek-ai/DeepSeek-R1 does not support it. Please open an issue on GitHubif you want it to be supported.

bao231 · 2025-02-04T06:53:47Z

v1 not support T4,are you support?

bao231 · 2025-02-04T08:36:01Z

@simon-mo

WoosukKwon · 2025-02-04T09:23:36Z

Hi @bao231, V1 does not support T4 or older-generation GPUs since the kernel libraries used in V1 (e.g., flash-attn) do not support them.

bao231 · 2025-02-04T10:41:17Z

V1 support other attention libs？has you plan? @WoosukKwon

robertgshaw2-redhat · 2025-02-04T14:44:23Z

I did a little experiment with DeepSeek-R1 on 8xH200 GPU.

vLLM 0.7.0 showed the following results with benchmark_serving.py --backend openai --base-url http://0.0.0.0:8000 --dataset-name=random --model deepseek-ai/DeepSeek-R1

with VLLM_USE_V1=1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [07:53<00:00,  2.11it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  473.62    
Total input tokens:                      1024000   
Total generated tokens:                  119550    
Request throughput (req/s):              2.11      
Output token throughput (tok/s):         252.42    
Total Token throughput (tok/s):          2414.51   
---------------Time to First Token----------------
Mean TTFT (ms):                          100636.33 
Median TTFT (ms):                        103588.53 
P99 TTFT (ms):                           197277.97 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          177.82    
Median TPOT (ms):                        172.14    
P99 TPOT (ms):                           363.05    
---------------Inter-token Latency----------------
Mean ITL (ms):                           173.08    
Median ITL (ms):                         136.46    
P99 ITL (ms):                            575.30    
==================================================

without VLLM_USE_V1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [05:24<00:00,  3.08it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  324.29    
Total input tokens:                      1024000   
Total generated tokens:                  119163    
Request throughput (req/s):              3.08      
Output token throughput (tok/s):         367.46    
Total Token throughput (tok/s):          3525.12   
---------------Time to First Token----------------
Mean TTFT (ms):                          29022.37  
Median TTFT (ms):                        32492.50  
P99 TTFT (ms):                           54457.59  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          125.16    
Median TPOT (ms):                        119.91    
P99 TPOT (ms):                           411.21    
---------------Inter-token Latency----------------
Mean ITL (ms):                           120.20    
Median ITL (ms):                         76.78     
P99 ITL (ms):                            656.11    
==================================================

In general, vLLM without VLLM_USE_V1 looked more productive. I also tried V0 with --request-rate 10 and got

Traffic request rate: 10.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [05:16<00:00,  3.16it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  316.20    
Total input tokens:                      1024000   
Total generated tokens:                  119448    
Request throughput (req/s):              3.16      
Output token throughput (tok/s):         377.76    
Total Token throughput (tok/s):          3616.21   
---------------Time to First Token----------------
Mean TTFT (ms):                          100122.09 
Median TTFT (ms):                        98699.05  
P99 TTFT (ms):                           201732.11 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          139.61    
Median TPOT (ms):                        104.30    
P99 TPOT (ms):                           1276.91   
---------------Inter-token Latency----------------
Mean ITL (ms):                           105.90    
Median ITL (ms):                         76.35     
P99 ITL (ms):                            648.36    
==================================================

Throughput was still 2 times lower than SGLang in the same benchmark. Today I updated vLLM to the new version (0.7.1) and decided to repeat the experiment. And the results in version V0 have become much better!

without VLLM_USE_V1 (with --request-rate 4)

Traffic request rate: 4.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [04:29<00:00,  3.71it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  269.74    
Total input tokens:                      1024000   
Total generated tokens:                  119805    
Request throughput (req/s):              3.71      
Output token throughput (tok/s):         444.14    
Total Token throughput (tok/s):          4240.35   
---------------Time to First Token----------------
Mean TTFT (ms):                          368.78    
Median TTFT (ms):                        269.07    
P99 TTFT (ms):                           3826.70   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          124.95    
Median TPOT (ms):                        122.03    
P99 TPOT (ms):                           214.93    
---------------Inter-token Latency----------------
Mean ITL (ms):                           123.32    
Median ITL (ms):                         75.30     
P99 ITL (ms):                            583.77    
==================================================

without VLLM_USE_V1 (with --request-rate 10)

Traffic request rate: 10.0
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
100%|█████████████████████████████████████████████████████████████| 1000/1000 [02:26<00:00,  6.83it/s]
============ Serving Benchmark Result ============
Successful requests:                     1000      
Benchmark duration (s):                  146.43    
Total input tokens:                      1024000   
Total generated tokens:                  119701    
Request throughput (req/s):              6.83      
Output token throughput (tok/s):         817.48    
Total Token throughput (tok/s):          7810.75   
---------------Time to First Token----------------
Mean TTFT (ms):                          14575.11  
Median TTFT (ms):                        13606.50  
P99 TTFT (ms):                           29954.96  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          297.01    
Median TPOT (ms):                        282.46    
P99 TPOT (ms):                           1393.69   
---------------Inter-token Latency----------------
Mean ITL (ms):                           262.67    
Median ITL (ms):                         132.89    
P99 ITL (ms):                            2840.40   
==================================================

But running vLLM with VLLM_USE_V1=1 I got en error TypeError: FlashAttentionImpl.__init__() got an unexpected keyword argument 'q_lora_rank' with previous warnings like

`torch.compile` is turned on, but the model deepseek-ai/DeepSeek-R1 does not support it. Please open an issue on GitHubif you want it to be supported.

Thanks!

We are aware of the performance gap for DeepSeekV3 and are actively working on it. See [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) #12676 which will resolve the gap. We will do a release hopefully today with this change
DeepSeekV3 is not yet supported on V1 since it requires chunked prefill. We are actively working on chunked prefill for MLA and hope to have it complete this week!

robertgshaw2-redhat · 2025-02-04T14:45:15Z

I'm either going insane, but with V1 qwen 8b instruct LLM just breaks in fp8 and around 25% of generations are just gibberish, with same running code and everything. Do I need to make a bug report, or it's an expected behaviour and I need some specific setup of sampling params for it to work in v1?

Can you provide a more detailed reproduction instruction?

cc @WoosukKwon

robertgshaw2-redhat · 2025-02-04T14:45:40Z

👍 I have not done a proper benchmark but V1 feels superior, i.e. higher throughput + lower latency, TTFT. The other thing that I have noticed is that logging has changed Running: 1 reqs, Waiting: 0 reqs, it used to print stats such token/s.
I have encountered a possible higher memory consumption issue, but am overall very pleased with the vllm community's hard work on V1. #12529

Thanks for fixing metrics logs in 0.7.1! Lack of pipeline parallelism in V1 is a show stopper for production deployments #11945

Thanks. We are actively working on PP

robertgshaw2-redhat · 2025-02-04T14:47:14Z

Maybe there is a better place to discuss this but the implementation for models that use more than one extra modality is quite non-intuitive. get_multimodal_embeddings() expects that we return a list or tensor of length equal to the number of multimodal items provided in the batch and we then have to make unintuitive assumptions on how the output passed into get_input_embeddings would look like because the batching being used while calling both functions is not the same. It would be much nicer if for example the input and output of get_multimodal_embeddings are dicts with the keys being the different modalities.

Check out #sig-multi-modality in our slack! This is the best place for a discussion like this

robertgshaw2-redhat · 2025-02-04T14:48:16Z

Something is weird with memory calculation in V1 and tensor parallel. Here are 2 cases that I tested recently:

vllm 0.7.0 on 2x A6000:

Starting normally a 32b-awq model and using --max-model-len 32768 --gpu-memory-utilization 0.98 --tensor-parallel 2 --max-num-batched-tokens 32768 --max-seq-len-to-capture 32768

Everything works as previously, GPUs both get to ~44-46GB usage

Using VLLM_USE_V1=1 and the exact same parameters as above:

GPUs both load up to ~24-25GB and it slowly goes up as inference runs. I've seen it go up to 32GB on each GPU.

Updating to vllm 0.7.1 and running a 7b-awq model this time, I also noticed that running the above command "normally" the logs show Maximum concurrency at 44x

Using V1 I get:

INFO 02-02 23:26:19 kv_cache_utils.py:400] Maximum concurrency for 32768 tokens per request: **22.25x**

And finally, with vllm 0.7.0 and 4x L4 loading a 32b-awq model with tp 4 works in "normal mode", but OOMs with V1.

Its pretty hard to follow what you are seeing. Please attach:

launch command
logs

Thanks!

gmonair · 2025-02-04T15:17:40Z

Its pretty hard to follow what you are seeing. Please attach:
* launch command

* logs

Hi, please see vllm_output(27)-OOM.log for OOM on 4x L4 and vllm_output(28)-WORKS.log to compare. The only difference between them is the V1 flag.

Launch command

my_env = os.environ.copy()
my_env["VLLM_USE_V1"] = "0"

# background task
command = [
    "python", 
    "-m", 
    "vllm.scripts", 
    "serve",
    "/kaggle/input/qwen25/transformers/r1-32b-awq/1",
    "--served-model-name", "model",
    "--tensor_parallel_size", "4",
    "--gpu_memory_utilization", "0.95",
    "--port", "9901",
    "--max-num-batched-tokens", "32768",
    "--max-seq-len-to-capture", "32768",
    "--max-model-len", "32768",
    "--enable_prefix_caching",
]

process = subprocess.Popen(command, stdout=log_file, stderr=log_file, env=my_env)

vllm_output(28)-WORKS.log
vllm_output(27)-OOM.log

caoyang-lqp · 2025-02-05T08:39:42Z

I ran the following code after upgrading the V1 version vllm and encountered an error:
import subprocess
import os
my_env = os.environ.copy()
my_env["VLLM_USE_V1"] = "1"
command = [
"python",
"-m",
"vllm.scripts",
"serve",
"./pretrained/intervl2-8B",
"--served-model-name", "intervl2-8B",
"--tensor_parallel_size", "2",
"--limit-mm-per-prompt","image=10" ,
"--pipeline-parallel-size","1",
"--gpu_memory_utilization", "0.9",
"--port", "40004",
"--max-num-batched-tokens", "10000",
"--max-seq-len-to-capture", "10000",
"--max-model-len", "10000",
"--enable_prefix_caching",
"--trust_remote_code"
]
process = subprocess.Popen(command, env=my_env)

However, if --tensor_parallel_size" is set to 1, it works fine. Is there a compatibility issue with the v1 version with the multi-card deployment model?

rstanislav · 2025-02-05T21:42:20Z

With dual rtx3090 in V1:
VLLM_USE_V1=1 REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt CUDA_DEVICE_ORDER=PCI_BUS_ID OMP_NUM_THREADS=1 CUDA_VISIBLE_DEVICES=0,1 vllm serve kosbu/QVQ-72B-Preview-AWQ --tensor-parallel-size 2 --gpu-memory-utilization 0.99 --api-key aaaaa --max-model-len 7000 --quantization=awq_marlin --enforce-eager

CUDA out of memory. Tried to allocate 594.00 MiB. GPU 0 has a total capacity of 23.48 GiB of which 587.38 MiB is free. Including non-PyTorch memory, this process has 22.89 GiB memory in use. Of the allocated memory 21.56 GiB is allocated by PyTorch, and 815.63 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation

With v0 it works, something changed about memory in V1.

JaheimLee · 2025-02-06T11:24:02Z

Will V1 support flashinfer in the future?

imkero · 2025-02-06T12:16:52Z

Does V1 support FP8 (W8A8) quantization?

I tried nm-testing/Qwen2-VL-7B-Instruct-FP8-dynamic on v0.7.1 V1 arch, no error thrown but got gibberish result. Same code and model works properly on v0.7.1 V0 arch.

UPDATE: it works on v0.7.1 V1 arch eager mode, but borken on v0.7.1 V1 arch torch.compiled mode. I'm figuring out if this problem is model-dependent or not.

UPDATE: tried another model nm-testing/DeepSeek-R1-Distill-Qwen-14B-FP8-Dynamic and same bug presents on v0.7.1 V1 arch torch.compiled mode

UPDATE: it works after i turned custom_ops on (change "none" to "all")

vllm/vllm/config.py

Lines 3237 to 3249 in 3ee696a

    
           if envs.VLLM_USE_V1 and self.model_config is not None and \ 
        
               not self.model_config.enforce_eager: 
        
               # NOTE(woosuk): Currently, we use inductor because the piecewise 
        
               # CUDA graphs do not work properly with the custom CUDA kernels. 
        
               # FIXME(woosuk): Disable inductor to reduce the compilation time 
        
               # and avoid any potential issues with the inductor. 
        
               self.compilation_config.custom_ops = ["none"] 
        
               self.compilation_config.use_cudagraph = True 
        
               self.compilation_config.use_inductor = True 
        
               self.compilation_config.cudagraph_num_of_warmups = 1 
        
               self.compilation_config.pass_config.enable_fusion = False 
        
               self.compilation_config.pass_config.enable_reshape = False 
        
               self.compilation_config.level = CompilationLevel.PIECEWISE

lyhh123 · 2025-02-08T08:41:15Z

When I tested the fine-tuned Qwen2.5_VL_3B model service using v1 mode (by setting the environment variable VLLM_USE_V1=1) and the default mode in OpenAI-compatible mode, I found inconsistencies in the output results.

I tested two samples:
• First sample: In v1 mode, the output was less than half of the expected result, while the default mode produced the complete output.
• Second sample: In v1 mode, the output was mostly complete but contained many obvious errors, whereas the default mode was correct and complete.

I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.

Has anyone else encountered a similar issue? If so, could this indicate a compatibility issue between v1 mode and Qwen2.5_VL_3B?

robertgshaw2-redhat · 2025-02-08T19:00:48Z

When I tested the fine-tuned Qwen2.5_VL_3B model service using v1 mode (by setting the environment variable VLLM_USE_V1=1) and the default mode in OpenAI-compatible mode, I found inconsistencies in the output results.

I tested two samples: • First sample: In v1 mode, the output was less than half of the expected result, while the default mode produced the complete output. • Second sample: In v1 mode, the output was mostly complete but contained many obvious errors, whereas the default mode was correct and complete.

I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.

Has anyone else encountered a similar issue? If so, could this indicate a compatibility issue between v1 mode and Qwen2.5_VL_3B?

cc @ywang96

ywang96 · 2025-02-08T19:02:03Z

When I tested the fine-tuned Qwen2.5_VL_3B model service using v1 mode (by setting the environment variable VLLM_USE_V1=1) and the default mode in OpenAI-compatible mode, I found inconsistencies in the output results.

I tested two samples: • First sample: In v1 mode, the output was less than half of the expected result, while the default mode produced the complete output. • Second sample: In v1 mode, the output was mostly complete but contained many obvious errors, whereas the default mode was correct and complete.

I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.

Has anyone else encountered a similar issue? If so, could this indicate a compatibility issue between v1 mode and Qwen2.5_VL_3B?

@lyhh123 can you open a separate issue for this and share some examples? There are multiple layers so I want to take a look where the issue might be.

I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.

This is also an interesting obversation since the V1-rearch for multimodal models should be model-agnostic, so I'm curious to see where the problem comes from.

lyhh123 · 2025-02-11T08:59:50Z

When I tested the fine-tuned Qwen2.5_VL_3B model service using v1 mode (by setting the environment variable VLLM_USE_V1=1) and the default mode in OpenAI-compatible mode, I found inconsistencies in the output results.
I tested two samples: • First sample: In v1 mode, the output was less than half of the expected result, while the default mode produced the complete output. • Second sample: In v1 mode, the output was mostly complete but contained many obvious errors, whereas the default mode was correct and complete.
I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.
Has anyone else encountered a similar issue? If so, could this indicate a compatibility issue between v1 mode and Qwen2.5_VL_3B?

@lyhh123 can you open a separate issue for this and share some examples? There are multiple layers so I want to take a look where the issue might be.

I conducted the same comparative experiment on Qwen2VL, and both v1 and default modes produced correct outputs.

This is also an interesting obversation since the V1-rearch for multimodal models should be model-agnostic, so I'm curious to see where the problem comes from.

Thank you for paying attention to my issue. Two days ago, I encountered this problem during testing. Over the past two days, I have made a series of attempts to adjust the sampling parameters, mainly by modifying top_p or other parameters to maintain output stability as much as possible.

Currently, I have re-tested using the v1/default model and the qwen2.5vl-3B model. Apart from content related to coordinates, the outputs have remained largely consistent. I attempted to adjust the parameters but was unable to reproduce the issue from two days ago.

I still remember that, with fixed parameters at that time, there were unexpected differences across multiple outputs between the v1 and default modes. However, I cannot rule out the possibility of other potential variables affecting the results at that time. I will do my best to identify the root cause of the issue, and if I make any relevant discoveries, I will update you promptly.

fan-niu · 2025-02-13T02:35:49Z

@robertgshaw2-redhat Hi now can we use V1 get high generate token throughput than V0 on Deepseek-R1?

WoosukKwon · 2025-02-13T05:36:45Z

@imkero Is the bug fixed now (without the change you suggested)? I wasn't able to reproduce the bug with the latest main.

imkero · 2025-02-13T08:45:10Z

@imkero Is the bug fixed now (without the change you suggested)? I wasn't able to reproduce the bug with the latest main.

@WoosukKwon

I have made a mistake. I found that it's use_cudagraph (but not custom_ops) that matters on this problem. I opened an issus #13212 to describe this problem in detail and for further discussion.

simon-mo added the misc label Jan 30, 2025

simon-mo changed the title ~~[V1] Feedback Threads~~ [V1] Feedback Thread Jan 30, 2025

simon-mo removed the misc label Jan 30, 2025

simon-mo pinned this issue Jan 30, 2025

njhill added the v1 label Feb 4, 2025

[V1] Feedback Thread #12568

[V1] Feedback Thread #12568

Comments

simon-mo commented Jan 30, 2025 • edited Loading

robertgshaw2-redhat commented Jan 30, 2025

wedobetter commented Jan 30, 2025 • edited Loading

m-harmonic commented Jan 30, 2025

robertgshaw2-redhat commented Jan 30, 2025

robertgshaw2-redhat commented Jan 30, 2025

dchichkov commented Jan 30, 2025

robertgshaw2-redhat commented Jan 31, 2025

hibukipanim commented Jan 31, 2025

akshay-loci commented Jan 31, 2025

robertgshaw2-redhat commented Jan 31, 2025

wedobetter commented Feb 2, 2025

Ouna-the-Dataweaver commented Feb 3, 2025

FrederickVu commented Feb 3, 2025

gmonair commented Feb 3, 2025 • edited Loading

Xarbirus commented Feb 3, 2025

bao231 commented Feb 4, 2025

bao231 commented Feb 4, 2025

WoosukKwon commented Feb 4, 2025

bao231 commented Feb 4, 2025

robertgshaw2-redhat commented Feb 4, 2025

robertgshaw2-redhat commented Feb 4, 2025

robertgshaw2-redhat commented Feb 4, 2025

robertgshaw2-redhat commented Feb 4, 2025

robertgshaw2-redhat commented Feb 4, 2025

gmonair commented Feb 4, 2025

caoyang-lqp commented Feb 5, 2025 • edited Loading

rstanislav commented Feb 5, 2025 • edited Loading

JaheimLee commented Feb 6, 2025

imkero commented Feb 6, 2025 • edited Loading

lyhh123 commented Feb 8, 2025

robertgshaw2-redhat commented Feb 8, 2025

ywang96 commented Feb 8, 2025 • edited Loading

lyhh123 commented Feb 11, 2025

fan-niu commented Feb 13, 2025

WoosukKwon commented Feb 13, 2025

imkero commented Feb 13, 2025

simon-mo commented Jan 30, 2025 •

edited

Loading

wedobetter commented Jan 30, 2025 •

edited

Loading

gmonair commented Feb 3, 2025 •

edited

Loading

caoyang-lqp commented Feb 5, 2025 •

edited

Loading

rstanislav commented Feb 5, 2025 •

edited

Loading

imkero commented Feb 6, 2025 •

edited

Loading

ywang96 commented Feb 8, 2025 •

edited

Loading