Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) #8029

Merged
merged 35 commits into from
Sep 5, 2024
Merged

[MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) #8029

merged 35 commits into from
Sep 5, 2024
+1,110 −208

Commits on Sep 1, 2024

  1. Enable stub mm image inputs for qwen models 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    de65d94 View commit details
    Browse the repository at this point in the history

  2. Add calc for max number of qwen image tokens 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    debcc8c View commit details
    Browse the repository at this point in the history

  3. Add unmodified visual qwen code, enable visual weight loading 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    9803fab View commit details
    Browse the repository at this point in the history

  4. Implement model processor for qwen, fix max tokens 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    364c110 View commit details
    Browse the repository at this point in the history

  5. Implement qwen input mapper and visual feature forward call 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    9b4eb9a View commit details
    Browse the repository at this point in the history

  6. Hacky integration of img pos / merging for qwen-vl 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    a709dd9 View commit details
    Browse the repository at this point in the history

  7. Add multimodal dummy data for qwen models 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    59339d2 View commit details
    Browse the repository at this point in the history

  8. Conditionally enable visual component to support qwen llm-only models 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    d6e3ca4 View commit details
    Browse the repository at this point in the history

  9. Fix hardcoded image start ID in image position extraction 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    8e27aa2 View commit details
    Browse the repository at this point in the history

  10. Enable chat for qwen-vl 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    f535d61 View commit details
    Browse the repository at this point in the history

  11. Improve validation, add qwen single image embed support 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    b10b73c View commit details
    Browse the repository at this point in the history

  12. Tentative support for multi-image embeddings in qwen 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    0611f19 View commit details
    Browse the repository at this point in the history

  13. Add example for qwen vl offline inference 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    541b7b5 View commit details
    Browse the repository at this point in the history

  14. run formatting and linting 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    1db0e6d View commit details
    Browse the repository at this point in the history

  15. Fix bug in image token input processing 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    ed3e15d View commit details
    Browse the repository at this point in the history

  16. Qwen - add comments, error handling in warmup 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    cadabb7 View commit details
    Browse the repository at this point in the history

  17. Fix device and dtype hack in Qwen resampler 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    4238c2f View commit details
    Browse the repository at this point in the history

  18. Update sequence data initialization 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    0258345 View commit details
    Browse the repository at this point in the history

  19. Flatten bn dimension for qwen 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    d0b8962 View commit details
    Browse the repository at this point in the history

  20. Switch qwen test to text only model 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    030b535 View commit details
    Browse the repository at this point in the history

  21. Run code formatting 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    27f819a View commit details
    Browse the repository at this point in the history

  22. Add image tag standardization, multimodal qwen tests 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    fcdd6f1 View commit details
    Browse the repository at this point in the history

  23. Remove support for <image> in qwen 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    a5c1201 View commit details
    Browse the repository at this point in the history

  24. Update docs for multimodal support 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    29a3c7f View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Update vllm/model_executor/models/qwen.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    7ac8ff9 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Add qwen back to llm docs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    3fe3a77 View commit details
    Browse the repository at this point in the history

  2. Make qwen/minicpmv embedding utils common 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    a4f5400 View commit details
    Browse the repository at this point in the history

  3. Make qwenvl / minicpmv2.0 resampler common 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    1989da8 View commit details
    Browse the repository at this point in the history

  4. Fix qwen warning for image placeholders 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    5e8409b View commit details
    Browse the repository at this point in the history

  5. Fix formatting, missing license, typehints 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    c889a69 View commit details
    Browse the repository at this point in the history

  6. Remove unreachable optional cross attn in qwenvl 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    2aa9549 View commit details
    Browse the repository at this point in the history

  7. Use parallel linear layers in qwenvl mlp 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    531d628 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Limit max_num_seqs

    @DarkLight1337
    DarkLight1337 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    d19418d View commit details
    Browse the repository at this point in the history

  2. Further limit

    @DarkLight1337
    DarkLight1337 committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    4f25926 View commit details
    Browse the repository at this point in the history

  3. Fix dummy data seq padding for multimodal qwen 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    00c2e09 View commit details
    Browse the repository at this point in the history