Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use non_reentrant_checkpoint fix requires_grad of input must be true for activation checkpoint layer in pipeline train. #4224

Merged
merged 36 commits into from
Sep 6, 2023
Merged

use non_reentrant_checkpoint fix requires_grad of input must be true for activation checkpoint layer in pipeline train. #4224

merged 36 commits into from
Sep 6, 2023

Commits on Aug 7, 2023

  1. feat: add non_reentrant_checkpoint

    @hughpu
    hughpu committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    a20c79c View commit details
    Browse the repository at this point in the history

  2. feat: add missing output postprocess and change the hook to record le… 

    …af forward tensor refs
    @hughpu
    hughpu committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    8aeba5f View commit details
    Browse the repository at this point in the history

  3. fix: make the multi_grad_hook registered after graph construction

    @hughpu
    hughpu committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    ee04fa8 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. fix: backward compatibility for multi_tensor_hook

    @hughpu
    hughpu committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    51f833d View commit details
    Browse the repository at this point in the history

  2. fix: nonlocal reference error of deepspeed_saved_tensors

    @hughpu
    hughpu committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    b29c1ef View commit details
    Browse the repository at this point in the history

  3. fix: reduce repeating hook registration

    @hughpu
    hughpu committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    37e7c23 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 9, 2023
    Configuration menu
    Copy the full SHA
    d7c5440 View commit details
    Browse the repository at this point in the history

  2. test: add test for `activation_checkpointing.checkpointing.non_reentr… 

    …ant_checkpoint`
    @hughpu
    hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    e22c487 View commit details
    Browse the repository at this point in the history

  3. Pass correct node size for ZeRO++ (microsoft#4085) 

    * Pass correct node size

* formatting

---------

Co-authored-by: Connor Holmes <development@cmikeh2.me>
Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
    @cmikeh2 @mrwyattii @hughpu
    3 people authored and hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    4d2a274 View commit details
    Browse the repository at this point in the history

  4. add deepspeed chat arxiv report (microsoft#4110) 

    * add deepspeed chat arxiv report

* add zeroquant v2 and fp

* add selective enhencement

* add ignore for 'Youn' in spell checker

---------

Co-authored-by: yaozhewei <zheweiy@berkeley.edu>
Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
    @conglongli @yaozhewei
    @mrwyattii @hughpu
    3 people authored and hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    d4d070b View commit details
    Browse the repository at this point in the history

  5. style: change flake8 detected style missmatch

    @hughpu
    hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    aaf309e View commit details
    Browse the repository at this point in the history

  6. test: hack to clone the test_activation_checkpointing module for re… 

    …use and add regression tests
    @hughpu
    hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    a910922 View commit details
    Browse the repository at this point in the history

  7. doc: explain the introduction of non_reentrant_checkpoint

    @hughpu
    hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    fc919b1 View commit details
    Browse the repository at this point in the history

  8. doc: explain the test of non_reentrant_checkpoint

    @hughpu
    hughpu committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    b6a0a44 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 9, 2023
    Configuration menu
    Copy the full SHA
    8ec86a4 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 9, 2023
    Configuration menu
    Copy the full SHA
    78c0d65 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 10, 2023
    Configuration menu
    Copy the full SHA
    e4eff23 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 11, 2023
    Configuration menu
    Copy the full SHA
    a6c7871 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 15, 2023
    Configuration menu
    Copy the full SHA
    fbbb760 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 16, 2023
    Configuration menu
    Copy the full SHA
    a338097 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 16, 2023
    Configuration menu
    Copy the full SHA
    a00cff1 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    c17cc3d View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 18, 2023
    Configuration menu
    Copy the full SHA
    a680399 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 19, 2023
    Configuration menu
    Copy the full SHA
    13e766d View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 21, 2023
    Configuration menu
    Copy the full SHA
    a46e326 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @hughpu
    hughpu authored Aug 23, 2023
    Configuration menu
    Copy the full SHA
    b5c03f4 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2023

  1. Merge branch 'master' into feat/non-reentrant-checkpoint

    @tjruwase
    tjruwase authored Aug 24, 2023
    Configuration menu
    Copy the full SHA
    13a026d View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. apply non_reentrant_checkpoint in pipeline parallel training

    @inkcherry
    inkcherry committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    0c18dda View commit details
    Browse the repository at this point in the history

  2. ut pass

    @inkcherry
    inkcherry committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    71421bf View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. Merge branch 'master' into use_reentrant

    @inkcherry
    inkcherry authored Aug 30, 2023
    Configuration menu
    Copy the full SHA
    18d64d3 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. fix ci

    @inkcherry
    inkcherry committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    d9edc63 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into use_reentrant

    @inkcherry
    inkcherry authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    d98790d View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. reduce check level for ci

    @inkcherry
    inkcherry committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    c27d334 View commit details
    Browse the repository at this point in the history

  2. reduce check level for ci

    @inkcherry
    inkcherry authored Sep 1, 2023
    Configuration menu
    Copy the full SHA
    79b427d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into use_reentrant

    @tohtana
    tohtana authored Sep 1, 2023
    Configuration menu
    Copy the full SHA
    80beb04 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Merge branch 'master' into use_reentrant

    @tohtana
    tohtana authored Sep 6, 2023
    Configuration menu
    Copy the full SHA
    bdc9db0 View commit details
    Browse the repository at this point in the history