[Core] Dynamic image size support for VLMs #5276
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DarkLight1337
changed the title
55 tasks
DarkLight1337
force-pushed
the
mm-image-tokenizer-2
branch
from
bdac3c9
to
25b5bb1
Compare
DarkLight1337
force-pushed
the
mm-image-tokenizer-2
branch
from
25b5bb1
to
9cfbcce
Compare
This was referenced Jun 5, 2024
ywang96
reviewed
Jul 2, 2024
DarkLight1337
force-pushed
the
mm-image-tokenizer-2
branch
from
93ad7de
to
cc540c3
Compare
ywang96
approved these changes
Jul 3, 2024
This was referenced Jul 3, 2024
prashantgupta24
pushed a commit
to opendatahub-io/vllm
that referenced
this pull request
Jul 3, 2024
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
robertgshaw2-neuralmagic
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Jul 7, 2024
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
xjpang
pushed a commit
to xjpang/vllm
that referenced
this pull request
Jul 8, 2024
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
xjpang
pushed a commit
to xjpang/vllm
that referenced
this pull request
Jul 24, 2024
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR uses the input registry introduced by #5214 to implement an input process that inserts image tokens automatically at the
LLMEnginelevel, so that it applies toLLM.generate.Accordingly, I have updated LLaVA-NeXT and Phi-3-Vision to support dynamic image size. Along the way, I have expanded the VLM tests to consider text-only and multiscale-image input in addition to the current single-scale image input.
Based on this, I have written a detailed guide on how to implement multimodal vLLM models.
Please note that this introduces a breaking change to users. Instead of manually repeating image tokens, the same prompt format as described in the corresponding HuggingFace repo should be used regardless of the model.
Related contributions
Follow-up to #5214.
This PR conflicts with #5237 as it inserts image tokens at the
OpenAIServinglevel. This PR has removed such logic from the server to avoid double insertion.