Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Fused MoE Marlin support for GPTQ #8217

Merged
merged 53 commits into from
Sep 10, 2024
Merged

[Misc] Fused MoE Marlin support for GPTQ #8217

merged 53 commits into from
Sep 10, 2024

Commits on Aug 30, 2024

  1. Enable 8-bit weights in Fused Marlin MoE

    @ElizaWszola
    ElizaWszola committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    0abac6f View commit details
    Browse the repository at this point in the history

  2. fix rocm

    @ElizaWszola
    ElizaWszola committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    fdf69c2 View commit details
    Browse the repository at this point in the history

  3. bad paste

    @ElizaWszola
    ElizaWszola committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    4da163b View commit details
    Browse the repository at this point in the history

  4. add test case; fix imports for tests

    @dsikka
    dsikka committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    21d2337 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into marlin-moe-8-bit

    @dsikka
    dsikka authored Aug 30, 2024
    Configuration menu
    Copy the full SHA
    080ab23 View commit details
    Browse the repository at this point in the history

  6. fix to adapt custom_routin_function

    @dsikka
    dsikka committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    638777a View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Use select_experts to compute top_k tensors in fused moe

    @ElizaWszola
    ElizaWszola committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    bd4b84d View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. bring back fused_moe_marlin -> fused_marlin_moe

    @ElizaWszola
    ElizaWszola committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    bef6b53 View commit details
    Browse the repository at this point in the history

  2. GPTQ Fused MoE class

    @ElizaWszola
    ElizaWszola committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    db1f07e View commit details
    Browse the repository at this point in the history

  3. Add GPTQMarlinMoEMethod to gptq_marlin.py

    @ElizaWszola
    ElizaWszola committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    6753789 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Use FusedMoE layer for all loads

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    7df4014 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into marlin-moe-8-bit

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    befc52b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'marlin-moe-8-bit' into gptq_fused_moe

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    c3dc249 View commit details
    Browse the repository at this point in the history

  4. Make sure that GPTQ runs through mixtral.py

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    2fa03e5 View commit details
    Browse the repository at this point in the history

  5. remove large model

    @dsikka
    dsikka committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    b45594c View commit details
    Browse the repository at this point in the history

  6. enforce float16A/scales for marlin moe

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    8a504d9 View commit details
    Browse the repository at this point in the history

  7. Cleanup, comments

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    effd2cd View commit details
    Browse the repository at this point in the history

  8. Merge branch 'marlin-moe-8-bit' into gptq_fused_moe

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    689ea0a View commit details
    Browse the repository at this point in the history

  9. cleanup

    @ElizaWszola
    ElizaWszola committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    ec47561 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. update/fix weight loading to support tp

    @dsikka
    dsikka committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    9f97b3b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. remove 8-bit stuff for now

    @ElizaWszola
    ElizaWszola committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    b841ac4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'gptq_fused_moe' of https://github.com/neuralmagic/vllm … 

    …into gptq_fused_moe
    @ElizaWszola
    ElizaWszola committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    a245032 View commit details
    Browse the repository at this point in the history

  3. fix; update large model testing cases

    @dsikka
    dsikka committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    9d8a80c View commit details
    Browse the repository at this point in the history

  4. add hack to support unfused mixtral pathway for int8

    @dsikka
    dsikka committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    315e22f View commit details
    Browse the repository at this point in the history

  5. fix install for tpu test

    @dsikka
    dsikka committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    565cc43 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. Move float16 typecast hack to gptq marlin moe method

    @ElizaWszola
    ElizaWszola committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    8886423 View commit details
    Browse the repository at this point in the history

  2. Move output type conversion to gptq method as well

    @ElizaWszola
    ElizaWszola committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    ab27497 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Enable 8-bit weights in Fused Marlin MoE

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    847e860 View commit details
    Browse the repository at this point in the history

  2. fix rocm

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    430a9cb View commit details
    Browse the repository at this point in the history

  3. bad paste

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    48047aa View commit details
    Browse the repository at this point in the history

  4. add test case; fix imports for tests

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    bfc4fae View commit details
    Browse the repository at this point in the history

  5. fix to adapt custom_routin_function

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    c5a2f62 View commit details
    Browse the repository at this point in the history

  6. Use select_experts to compute top_k tensors in fused moe

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    2b308c4 View commit details
    Browse the repository at this point in the history

  7. bring back fused_moe_marlin -> fused_marlin_moe

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    71256d4 View commit details
    Browse the repository at this point in the history

  8. GPTQ Fused MoE class

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    7aa844c View commit details
    Browse the repository at this point in the history

  9. Add GPTQMarlinMoEMethod to gptq_marlin.py

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    0f7bec3 View commit details
    Browse the repository at this point in the history

  10. Use FusedMoE layer for all loads

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    cb0001e View commit details
    Browse the repository at this point in the history

  11. Make sure that GPTQ runs through mixtral.py

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    33090a3 View commit details
    Browse the repository at this point in the history

  12. enforce float16A/scales for marlin moe

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d479837 View commit details
    Browse the repository at this point in the history

  13. remove large model

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    8baaec6 View commit details
    Browse the repository at this point in the history

  14. Cleanup, comments

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    8fbc181 View commit details
    Browse the repository at this point in the history

  15. cleanup

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    839915f View commit details
    Browse the repository at this point in the history

  16. remove 8-bit stuff for now

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a5bc626 View commit details
    Browse the repository at this point in the history

  17. update/fix weight loading to support tp

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    c573fa1 View commit details
    Browse the repository at this point in the history

  18. fix; update large model testing cases

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a991d82 View commit details
    Browse the repository at this point in the history

  19. add hack to support unfused mixtral pathway for int8

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d57804d View commit details
    Browse the repository at this point in the history

  20. fix install for tpu test

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    96fa486 View commit details
    Browse the repository at this point in the history

  21. Move float16 typecast hack to gptq marlin moe method

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1faab90 View commit details
    Browse the repository at this point in the history

  22. Move output type conversion to gptq method as well

    @ElizaWszola @dsikka
    ElizaWszola authored and dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    970e06a View commit details
    Browse the repository at this point in the history

  23. typo fix; fix comment

    @dsikka
    dsikka committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    fd0a4f2 View commit details
    Browse the repository at this point in the history

  24. Merge branch 'gptq_fused_moe' of https://github.com/neuralmagic/vllm … 

    …into gptq_fused_moe
    @ElizaWszola
    ElizaWszola committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    3ac9273 View commit details
    Browse the repository at this point in the history

  25. Clarify comment, change how we process bias

    @ElizaWszola
    ElizaWszola committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d51a2f4 View commit details
    Browse the repository at this point in the history

  26. Merge branch 'main' into gptq_fused_moe

    @dsikka
    dsikka authored Sep 9, 2024
    Configuration menu
    Copy the full SHA
    12f05c5 View commit details
    Browse the repository at this point in the history