Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Zigzag Ring attention #5905

Merged
merged 37 commits into from
Aug 16, 2024
Merged

[Feature] Zigzag Ring attention #5905

merged 37 commits into from
Aug 16, 2024

Commits on Aug 8, 2024

  1. halfway

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    519818f View commit details
    Browse the repository at this point in the history

  2. fix cross-PP-stage position id length diff bug

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    04b14a2 View commit details
    Browse the repository at this point in the history

  3. fix typo

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    45b9ac1 View commit details
    Browse the repository at this point in the history

  4. fix typo

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    3047c4e View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @Edenzzzz
    pre-commit-ci[bot] authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    c0a5048 View commit details
    Browse the repository at this point in the history

  6. unified cross entropy func for all shardformer models

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    748b0a1 View commit details
    Browse the repository at this point in the history

  7. remove redundant lines

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    0262e6b View commit details
    Browse the repository at this point in the history

  8. add basic ring attn; debug cross entropy

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    7dfdac1 View commit details
    Browse the repository at this point in the history

  9. fwd bwd logic complete

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    a4d4e6a View commit details
    Browse the repository at this point in the history

  10. fwd bwd logic complete; add experimental triton rescale

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    7a4e284 View commit details
    Browse the repository at this point in the history

  11. precision tests passed

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    f8be40d View commit details
    Browse the repository at this point in the history

  12. precision tests passed

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    c3d7a86 View commit details
    Browse the repository at this point in the history

  13. fix typos and remove misc files

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    313bc48 View commit details
    Browse the repository at this point in the history

  14. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @Edenzzzz
    pre-commit-ci[bot] authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    98627e8 View commit details
    Browse the repository at this point in the history

  15. add sp_mode to benchmark; fix varlen interface

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    a3bb451 View commit details
    Browse the repository at this point in the history

  16. update softmax_lse shape by new interface

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    b104530 View commit details
    Browse the repository at this point in the history

  17. change tester name

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    6cbc5f6 View commit details
    Browse the repository at this point in the history

  18. remove buffer clone; support packed seq layout

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    bf00238 View commit details
    Browse the repository at this point in the history

  19. add varlen tests

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    bd2d642 View commit details
    Browse the repository at this point in the history

  20. fix typo

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    ea11927 View commit details
    Browse the repository at this point in the history

  21. all tests passed

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    2f8e188 View commit details
    Browse the repository at this point in the history

  22. add dkv_group; fix mask

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    919eff5 View commit details
    Browse the repository at this point in the history

  23. remove debug statements

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    04d2f88 View commit details
    Browse the repository at this point in the history

  24. add comments

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    392bde6 View commit details
    Browse the repository at this point in the history

  25. q1 index only once

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e760507 View commit details
    Browse the repository at this point in the history

  26. remove events to simplify stream sync

    @Edenzzzz
    Edenzzzz authored and Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e90e984 View commit details
    Browse the repository at this point in the history

  27. clarify kv_comm.wait()

    @Edenzzzz
    Edenzzzz committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e26c910 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. use torch.compile; add nsys

    Edenzzzz committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    b6b2333 View commit details
    Browse the repository at this point in the history

  2. simplify forward/backward logic

    Edenzzzz committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    d3831b4 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. 2d ring forward passed

    Edenzzzz committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    0094bc0 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. 2d ring backward passed

    Edenzzzz committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    581ec0f View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. fixes

    Edenzzzz committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    1344849 View commit details
    Browse the repository at this point in the history

  2. fix ring attn loss

    Edenzzzz committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    e6bcde2 View commit details
    Browse the repository at this point in the history

  3. 2D ring backward + llama passed

    Edenzzzz committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    b4c0809 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. follow conventions

    Edenzzzz committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    26b008e View commit details
    Browse the repository at this point in the history

  2. fix dist logger

    Edenzzzz committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    a68dd2f View commit details
    Browse the repository at this point in the history

  3. add a manual inner ring size option

    Edenzzzz committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    be5fed5 View commit details
    Browse the repository at this point in the history