Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No-ZeRO reshaping #289

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

No-ZeRO reshaping #289

wants to merge 37 commits into from

Commits on Jan 24, 2022

  1. Reshape deepspeed checkpoint

    @tjruwase
    tjruwase committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    67c08f0 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into ds_ckpt_reshape

    @stas00
    stas00 committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    fec1ec5 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2022

  1. add checkpoint tests

    @stas00
    stas00 committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    675f12c View commit details
    Browse the repository at this point in the history

  2. Validate input folder

    @tjruwase
    tjruwase committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    e379065 View commit details
    Browse the repository at this point in the history

  3. Tests for tp/pp reshape

    @tjruwase
    tjruwase committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    a1068e4 View commit details
    Browse the repository at this point in the history

  4. remove debug folders

    @stas00
    stas00 committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    115bd31 View commit details
    Browse the repository at this point in the history

  5. fix test_checkpoint_reshaping_empty_dir

    @stas00
    stas00 committed Jan 25, 2022
    Configuration menu
    Copy the full SHA
    cc2fad1 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2022

  1. Fix unit tests

    @tjruwase
    tjruwase committed Jan 26, 2022
    Configuration menu
    Copy the full SHA
    b6733d5 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2022

  1. Remove deepspeed checkpoint utils

    @tjruwase
    tjruwase committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    9bf7ac5 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. Use DS 3D reshaping utils

    @tjruwase
    tjruwase committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    29ca2bc View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. Merge remote-tracking branch 'origin/main' into ds_ckpt_reshape

    @stas00
    stas00 committed May 18, 2022
    Configuration menu
    Copy the full SHA
    a3ef778 View commit details
    Browse the repository at this point in the history

  2. convert to bf16

    @stas00
    stas00 committed May 18, 2022
    Configuration menu
    Copy the full SHA
    6d86358 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2022

  1. wip universal chkpt

    @stas00
    stas00 committed May 20, 2022
    Configuration menu
    Copy the full SHA
    804b497 View commit details
    Browse the repository at this point in the history

  2. rename

    @stas00
    stas00 committed May 20, 2022
    Configuration menu
    Copy the full SHA
    c29d336 View commit details
    Browse the repository at this point in the history

  3. rename

    @stas00
    stas00 committed May 20, 2022
    Configuration menu
    Copy the full SHA
    9c44793 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2022

  1. wip on fragments dealing

    @stas00
    stas00 committed May 24, 2022
    Configuration menu
    Copy the full SHA
    7e0a81b View commit details
    Browse the repository at this point in the history

  2. cleanup

    @stas00
    stas00 committed May 24, 2022
    Configuration menu
    Copy the full SHA
    d300512 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2022

  1. Loading universal checkpoint with reshaping

    @tjruwase
    tjruwase committed May 26, 2022
    Configuration menu
    Copy the full SHA
    ab0a7f8 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2022

  1. all gpu1<->2 reshapes work

    @stas00
    stas00 committed May 28, 2022
    Configuration menu
    Copy the full SHA
    d5e33de View commit details
    Browse the repository at this point in the history

Commits on May 31, 2022

  1. param attrs

    @tjruwase
    tjruwase committed May 31, 2022
    Configuration menu
    Copy the full SHA
    85ff56c View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2022

  1. make the tests adaptable to the number of available gpus

    @stas00
    stas00 committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    f01fa4a View commit details
    Browse the repository at this point in the history

  2. WIP

    @tjruwase
    tjruwase committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    f29bacc View commit details
    Browse the repository at this point in the history

  3. WIP

    @tjruwase
    tjruwase committed Jun 1, 2022
    Configuration menu
    Copy the full SHA
    dd0aeb6 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2022

  1. WIP

    @tjruwase
    tjruwase committed Jun 2, 2022
    Configuration menu
    Copy the full SHA
    3bf14fd View commit details
    Browse the repository at this point in the history

  2. WIP

    @tjruwase
    tjruwase committed Jun 2, 2022
    Configuration menu
    Copy the full SHA
    7ae002d View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2022

  1. Debug functions

    @tjruwase
    tjruwase committed Jun 4, 2022
    Configuration menu
    Copy the full SHA
    55bb514 View commit details
    Browse the repository at this point in the history

  2. args should be required, don't create another latest file

    @stas00
    stas00 committed Jun 4, 2022
    Configuration menu
    Copy the full SHA
    795fedb View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2022

  1. Parallelize shard extraction

    @tjruwase
    tjruwase committed Jun 6, 2022
    Configuration menu
    Copy the full SHA
    cc8810b View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2022

  1. close+join pool; add tqdm; comment out noise

    @stas00
    stas00 committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    04d9ad0 View commit details
    Browse the repository at this point in the history

  2. rename

    @stas00
    stas00 committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    bca5af4 View commit details
    Browse the repository at this point in the history

  3. parameterize

    @stas00
    stas00 committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    721380b View commit details
    Browse the repository at this point in the history

  4. Parallel slice merging

    @tjruwase
    tjruwase committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    e8a1ccf View commit details
    Browse the repository at this point in the history

  5. Cleanup

    @tjruwase
    tjruwase committed Jun 7, 2022
    Configuration menu
    Copy the full SHA
    a247614 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2022

  1. Merge branch 'main' into nozero_reshape

    @Muennighoff
    Muennighoff committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    d94e1e2 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2022

  1. Add many-to-many non-ZeRO reshaping

    @Muennighoff
    Muennighoff committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    c14df23 View commit details
    Browse the repository at this point in the history

  2. Explain many-to-many status

    @Muennighoff
    Muennighoff committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    e8ba66a View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2022

  1. Merge main

    @Muennighoff
    Muennighoff committed Jul 4, 2022
    Configuration menu
    Copy the full SHA
    86c74e6 View commit details
    Browse the repository at this point in the history