Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync layer norm #271

Draft
wants to merge 38 commits into
base: thomas/test_different_layer_norm
Choose a base branch
from
Draft

Sync layer norm #271

wants to merge 38 commits into from

Commits on Mar 24, 2022

  1. Better

    @thomasw21
    thomasw21 committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    07ccb3d View commit details
    Browse the repository at this point in the history

  2. Force synchronize the layer norms parameters across all TP

    @thomasw21
    thomasw21 committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    391ed48 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2022

  1. import mpu

    @stas00
    stas00 authored Mar 25, 2022
    Configuration menu
    Copy the full SHA
    98d0e7c View commit details
    Browse the repository at this point in the history

  2. use the bf16 branch for testing

    @stas00
    stas00 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    279a77e View commit details
    Browse the repository at this point in the history

  3. torch.testing.assert_equal didn't make it (#273) 

    looks like pt-1.11 dropped `torch.testing.assert_equal`, so using `torch.testing.assert_equal` instead
    @stas00
    stas00 authored Mar 25, 2022
    Configuration menu
    Copy the full SHA
    87a9dba View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into thomas/fix_layer_norm

    @stas00
    stas00 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    dbb5914 View commit details
    Browse the repository at this point in the history

  5. bf16 comms requite pt-1.11

    @stas00
    stas00 authored Mar 25, 2022
    Configuration menu
    Copy the full SHA
    70f91f8 View commit details
    Browse the repository at this point in the history

  6. already part of the function

    @stas00
    stas00 authored Mar 25, 2022
    Configuration menu
    Copy the full SHA
    835a3e5 View commit details
    Browse the repository at this point in the history

  7. reproduce the crashing on resume

    @stas00
    stas00 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    37795a9 View commit details
    Browse the repository at this point in the history

  8. run just the test we want for now

    @stas00
    stas00 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    3ec65f7 View commit details
    Browse the repository at this point in the history

  9. all_reduce is an in_place operation

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    8271d41 View commit details
    Browse the repository at this point in the history

  10. Make a test that TP reshaping works

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    b418b47 View commit details
    Browse the repository at this point in the history

  11. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    4b7207b View commit details
    Browse the repository at this point in the history

  12. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    3bc5824 View commit details
    Browse the repository at this point in the history

  13. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    05c99db View commit details
    Browse the repository at this point in the history

  14. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    55e10c6 View commit details
    Browse the repository at this point in the history

  15. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    2ab8a3a View commit details
    Browse the repository at this point in the history

  16. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    d357839 View commit details
    Browse the repository at this point in the history

  17. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    5fb231c View commit details
    Browse the repository at this point in the history

  18. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    cc7ff45 View commit details
    Browse the repository at this point in the history

  19. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    7cdb1be View commit details
    Browse the repository at this point in the history

  20. Fix load issue

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    4574ec9 View commit details
    Browse the repository at this point in the history

  21. Woops

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    04e89d1 View commit details
    Browse the repository at this point in the history

  22. Fix checkpoint path

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    e943100 View commit details
    Browse the repository at this point in the history

  23. Test that force sync will allow TP changes

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    09cead3 View commit details
    Browse the repository at this point in the history

  24. Nit

    @thomasw21
    thomasw21 committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    77abee6 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2022

  1. Now that we have a force sync mechanism, let's try to reproduce

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    64a62c8 View commit details
    Browse the repository at this point in the history

  2. Compare model_states_rank

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    0b7afcc View commit details
    Browse the repository at this point in the history

  3. test

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    ce01733 View commit details
    Browse the repository at this point in the history

  4. Row column bias should be synchronized as well

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    89ab0b7 View commit details
    Browse the repository at this point in the history

  5. New list of matching embeddings

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    42997b2 View commit details
    Browse the repository at this point in the history

  6. Figure out why state differs

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    e0ef168 View commit details
    Browse the repository at this point in the history

  7. Test for final weight

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    1fc4fe8 View commit details
    Browse the repository at this point in the history

  8. Test that torch_rng_state

    @thomasw21
    thomasw21 committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    7ebbed1 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. Fix non matching torch_rng_state for tp_rank=0

    @thomasw21
    thomasw21 committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    2c49216 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2022

  1. Update test

    @thomasw21
    thomasw21 committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    007ecb4 View commit details
    Browse the repository at this point in the history

  2. I'm surprised one can apply inplace operation here

    @thomasw21
    thomasw21 committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    c3844b5 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2022

  1. Test out the loss from the fp32 weights and optimizer states

    @thomasw21
    thomasw21 committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    189f054 View commit details
    Browse the repository at this point in the history