[Model] Initialize Phi-3-vision support

[Isotr0py]

FILL IN THE PR DESCRIPTION HERE

FIX #4958

• [Feature]: microsoft/Phi-3-vision-128k-instruct Vision support #4958

Note that I only implemented the Phi-3-vision model and only tested the model weights loading currently.

This PR is still under very low completion.

• Implement the Phi-3-vision model and weights loading logics
• Test Phi-3-vision inference and add Phi3ImageInputs etc.
• Add Phi-3-vision ImageProcessor (waiting for [Core] Support image processor #4197 merged)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Model] for adding a new model or improving an existing model. Model name should appear in the title.
• [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
• [Kernel] for changes affecting CUDA kernels or other compute kernels.
• [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
• [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• We adhere to Google Python style guide and Google C++ style guide.
• Pass all linter checks. Please use format.sh to format your code.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
• Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

• After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
• After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
• After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
• Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

[Reichenbachian]

Woot woot! I love the fast movement on this.

[Isotr0py]

Well, it seems that the inference can work now:

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 05-23 17:49:28 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/data/LLM-model/Phi-3-vision-128k-instruct', speculative_config=None, tokenizer='/data/LLM-model/Phi-3-vision-128k-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/data/LLM-model/Phi-3-vision-128k-instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 05-23 17:49:28 cpu_executor.py:116] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 05-23 17:49:28 cpu_executor.py:143] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 05-23 17:49:28 selector.py:62] Using Torch SDPA backend.
INFO 05-23 17:49:43 phi3v.py:104] learnable separator enabled for hd transform, hd_transform_order = sub_glb
INFO 05-23 17:49:48 cpu_executor.py:72] # CPU blocks: 682
Processed prompts:   0%|                                                                                                                            | 0/1 [00:00<?, ?it/s, Generation Speed: 0.00 toks/s]torch.Size([1, 17, 3, 336, 336])
INFO 05-23 17:50:20 phi3v.py:263] img_embeds size: torch.Size([1, 17, 3, 336, 336]), image sizes: (1008, 1344) loading time 0:00:29.625456
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [01:48<00:00, 108.67s/it, Generation Speed: 0.15 toks/s]
 The image shows a city street corner with a prominent red stop sign.

[Reichenbachian]

Hype. How can I help out? Would love to push the ball forward on this.

[ywang96]

Hey @Isotr0py - Now that we have done some refactoring around VLMs, do you have some bandwidth to update this PR so I can start reviewing? Thanks!

[Isotr0py]

@ywang96 I have updated the code to make examples/phi3v_example.py work now. Can you have a look at this?

[DarkLight1337]

Can you add a test to check the model's consistency with its HuggingFace counterpart? That would be the most straightforward way to verify the correctness of your implementation.

[Youho99]

I am interested in using this phi3-v implementation

When do you think this will be functional / added to the vLLM project officially?

[DarkLight1337]

Okay Thank You...Do you know about Multiple Models support? Can vLLM serving multiple models?

No, that is not supported. You can run multiple vLLM instances at the same time though to achieve a similar result.

[sung-ho-moon]

I understand. thank you

[Youho99]

Hello guys!

I installed one of the latest commits supporting Phi3-vision, to test it, to potentially use it later.

The implementation is functional.
However, it is impossible for me to have a long answer.

I ask the VLM to describe an image (a graph in my case).
He answers me, but only the beginning of his answer.

Example: “The image shows a line graph representing the projected share of the population in extreme”

Impossible to get the rest of the answer.

This is definitely related to the "max_tokens", so I increased it (tested with 1024/2048/4096), but I still get the same result.

Do you have any idea how to go about getting the full answer?

Thanks!

[Isotr0py]

@Youho99 Have you limited the max_model_len? This may also affect the length of result.

Though phi3-vision in VLLM has numeric difference issue currently, it's unlikely that it would output incomplete result...

[Youho99]

@Isotr0py max_model_len=4096

I take the phi3v_example.py as base for my code.

Actually, I don't think it "generates an incomplete result" per se. I rather think that a (complete) response is generated, but that the display of the response, for some unknown reason, only gives the beginning of the response.
(this is just a hypothesis)

[Isotr0py]

Can you remove or increase the value in the max_model_len=4096 to see if it can give full answer?

max_model_len=4096 is used to prevent OOM with 128k context_length on my device and I forgot to remove it ...

[Youho99]

Same results with max_model_len=8192, max_model_len=16384
OOM with max_model_len=32768

That's my code :

import os
import subprocess
from PIL import Image
from vllm import LLM, SamplingParams
from vllm.multimodal.image import ImagePixelData
import torch
import gc

model = None

def initialize_phi3v(model_path="microsoft/Phi-3-vision-128k-instruct"):

    global model

    # Delete previous model if it exists
    if 'model' in globals() and model is not None:
        del model
        gc.collect()
        if torch.cuda.is_available():
            torch.cuda.empty_cache()

    model = LLM(
        model=model_path,
        trust_remote_code=True,
        max_model_len=4096,
        image_input_type="pixel_values",
        image_token_id=32044,
        image_input_shape="1,3,1008,1344",
        image_feature_size=1921,
        disable_image_processor=False,
        disable_sliding_window=True, # For using Flash-Attn
    )
    return model

def run_phi3v(model, image_path, prompt):
    image = Image.open(image_path)

    # single-image prompt
    prompt = f"<|user|>\n<|image_1|>\n{prompt}<|end|>\n<|assistant|>\n"  # noqa: E501
    prompt = prompt.replace("<|image_1|>", "<|image|>" * 1921 + "<s>")

    sampling_params = SamplingParams(temperature=0, max_tokens=2048)

    outputs = model.generate({
        "prompt": prompt,
        "sampling_params": sampling_params,
        "multi_modal_data": ImagePixelData(image),
    })
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

model = initialize_phi3v()

image_path = "images/00006834003066.png"
prompt = "What is shown in this image?"

run_phi3v(model, image_path, prompt)

[Isotr0py]

@Youho99 Seems that the sampling_params is passed incorrectly. This should work:

outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": ImagePixelData(image),
    },
    sampling_params=sampling_params)

[Youho99]

@Youho99 Seems that the sampling_params is passed incorrectly. This should work:

outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": ImagePixelData(image),
    },
    sampling_params=sampling_params)

That's works! Thanks!

[subodhchhabra]

Getting an error when trying to scale with ray(tensor_parallel_size =2) , any idea what is happening here ?

Traceback (most recent call last):
File "/tmp/ray/session_2024-06-20_09-38-29_030934_8/runtime_resources/working_dir_files/_ray_pkg_0cbd81c426d5525d/src/phi3v_ext.py", line 94, in main
parse_result(ds)
File "/tmp/ray/session_2024-06-20_09-38-29_030934_8/runtime_resources/working_dir_files/_ray_pkg_0cbd81c426d5525d/src/phi3v_ext.py", line 50, in parse_result
for output in ds.iter_rows():
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/iterator.py", line 245, in _wrapped_iterator
for batch in batch_iterable:
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/iterator.py", line 162, in _create_iterator
block_iterator, stats, blocks_owned_by_consumer = self._to_block_iterator()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/iterator/iterator_impl.py", line 33, in _to_block_iterator
block_iterator, stats, executor = ds._plan.execute_to_iterator()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/exceptions.py", line 86, in handle_trace
raise e.with_traceback(None) from SystemException()
ray.exceptions.ActorDiedError: The actor died because of an error raised in its creation task, �[36mray::_MapWorker.init()�[39m (pid=812, ip=10.89.133.29, actor_id=2330fca6520bd378b441638d05000000, repr=MapWorker(MapBatches(LLMImagePredictor)))
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/actor_pool_map_operator.py", line 350, in init
self._map_transformer.init()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/operators/map_transformer.py", line 126, in init
self._init_fn()
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/planner/plan_udf_map_op.py", line 118, in init_fn
ray.data._cached_fn = op_fn(
File "/home/ray/anaconda3/lib/python3.10/site-packages/ray/data/_internal/execution/util.py", line 70, in init
super().init(*args, **kwargs)
File "/tmp/ray/session_2024-06-20_09-38-29_030934_8/runtime_resources/working_dir_files/_ray_pkg_0cbd81c426d5525d/src/image_predictor.py", line 34, in init
self.llm = LLM(
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in init
self.llm_engine = LLMEngine.from_engine_args(
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 384, in from_engine_args
engine = cls(
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 230, in init
self.model_executor = executor_class(
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in init
super().init(*args, **kwargs)
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in init
self._init_executor()
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 40, in _init_executor
self._init_workers_ray(placement_group)
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 185, in _init_workers_ray
self._run_workers("init_worker", all_kwargs=init_worker_all_kwargs)
File "/home/ray/anaconda3/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 282, in _run_workers
ray_worker_outputs = ray.get(ray_worker_outputs)
ray.exceptions.RayTaskError(RaySystemError): �[36mray::RayWorkerWrapper.execute_method()�[39m (pid=780, ip=10.89.142.145, actor_id=060bebe8f7ccbb9727b485db05000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7fde9b6909a0>)
At least one of the input arguments for this task could not be computed:
ray.exceptions.RaySystemError: System error: No module named 'transformers_modules'
traceback: Traceback (most recent call last):
ModuleNotFoundError: No module named 'transformers_modules'

[DarkLight1337]

May be related to #4169 #3593.