Code

FSDP integration#6152

Code

Closed

SeanNaren wants to merge 77 commits intomasterLightning-AI/pytorch-lightning:masterfrom feat/fsdpLightning-AI/pytorch-lightning:feat/fsdpCopy head branch name to clipboard

+581-13

Commits

Commits on Feb 23, 2021

• Add initial FSDP integration

  

  SeanNaren

  committed
  78f1eb4

  Copy full SHA for 78f1eb4

• Fix error in refactor

  

  SeanNaren

  committed
  c36e00a

  Copy full SHA for c36e00a

Commits on Feb 24, 2021

• update

  tchaton

  committed
  59dbb83

  Copy full SHA for 59dbb83

• Revert "update"

  Show description for 19a1440
  

  SeanNaren

  committed
  19a1440

  Copy full SHA for 19a1440

• Address reviews

  

  SeanNaren

  committed
  3b38615

  Copy full SHA for 3b38615

• Fix doc string

  

  SeanNaren

  committed
  5ff06ab

  Copy full SHA for 5ff06ab

• Even moar code review

  

  SeanNaren

  committed
  36434f0

  Copy full SHA for 36434f0

• Add deprecation

  

  SeanNaren

  committed
  c61a190

  Copy full SHA for c61a190

Commits on Feb 25, 2021

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  1c4f011

  Copy full SHA for 1c4f011

• Fix name of test

  

  SeanNaren

  committed
  02599e6

  Copy full SHA for 02599e6

Commits on Mar 1, 2021

• Integrate nesting, fix bugs across implementation

  

  SeanNaren

  committed
  e79977a

  Copy full SHA for e79977a

Commits on Mar 2, 2021

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  d15d4b5

  Copy full SHA for d15d4b5

• Formatting types

  

  SeanNaren

  committed
  ebf1818

  Copy full SHA for ebf1818

• Add additional tests for accelerator model

  

  SeanNaren

  committed
  290e8fd

  Copy full SHA for 290e8fd

• Fix import

  

  SeanNaren

  committed
  5c5f762

  Copy full SHA for 5c5f762

Commits on Mar 3, 2021

• Few test fixes, expose params

  

  SeanNaren

  committed
  d28438b

  Copy full SHA for d28438b

• Allow training_type_plugin to delay optimizer configure

  

  SeanNaren

  committed
  ab591a8

  Copy full SHA for ab591a8

• Merge branch 'feat/fsdp_2n' into feat/fsdp

  

  SeanNaren

  committed
  23ccdb8

  Copy full SHA for 23ccdb8

• Add missing references to trainer, add a CPU accelerator based test

  

  SeanNaren

  committed
  a60f2c0

  Copy full SHA for a60f2c0

Commits on Mar 4, 2021

• Merge branch 'feat/fsdp_2n' into feat/fsdp

  

  SeanNaren

  committed
  3d4e6df

  Copy full SHA for 3d4e6df

Commits on Mar 9, 2021

• Update for latest API changes to fairscale

  

  SeanNaren

  committed
  516bd04

  Copy full SHA for 516bd04

Commits on Mar 23, 2021

• Add base hook for model parallel

  

  SeanNaren

  committed
  9f8864f

  Copy full SHA for 9f8864f

Commits on Mar 25, 2021

• fix callback signature

  kaushikb11

  committed
  eac5344

  Copy full SHA for eac5344

• Simplify hook

  

  SeanNaren

  committed
  32df0cb

  Copy full SHA for 32df0cb

• Add hook logic

  

  SeanNaren

  committed
  282a133

  Copy full SHA for 282a133

• add tests

  kaushikb11

  committed
  7a94e72

  Copy full SHA for 7a94e72

• add property setter

  kaushikb11

  committed
  8091481

  Copy full SHA for 8091481

• add logic for being called once

  kaushikb11

  committed
  633fc77

  Copy full SHA for 633fc77

• Update changelog

  kaushikb11

  committed
  c99a36f

  Copy full SHA for c99a36f

• Merge branch 'master' into feat/model_parallel_hook

  kaushikb11

  authored
  a68c8d7

  Copy full SHA for a68c8d7

• Fix

  kaushikb11

  committed
  9529a22

  Copy full SHA for 9529a22

• fix return type

  kaushikb11

  committed
  3c1c782

  Copy full SHA for 3c1c782

• Merge branch 'master' into feat/fsdp

  Show description for 7daba43
  

  SeanNaren

  committed
  7daba43

  Copy full SHA for 7daba43

• Fix property name

  

  SeanNaren

  committed
  87ec222

  Copy full SHA for 87ec222

• Merge branch 'feat/model_parallel_hook' into feat/fsdp

  Show description for 966b2e5
  

  SeanNaren

  committed
  966b2e5

  Copy full SHA for 966b2e5

• Updaet wrapper, use latest fixes for hooks

  

  SeanNaren

  committed
  5f6e039

  Copy full SHA for 5f6e039

• Swap hook order

  

  SeanNaren

  committed
  b512e72

  Copy full SHA for b512e72

Commits on Mar 29, 2021

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  8ba82df

  Copy full SHA for 8ba82df

• Small changes

  

  SeanNaren

  committed
  1e5ca37

  Copy full SHA for 1e5ca37

• Fixes

  

  SeanNaren

  committed
  936dc1a

  Copy full SHA for 936dc1a

Commits on Apr 1, 2021

• Remove activation checkpointing

  

  SeanNaren

  committed
  a6de18e

  Copy full SHA for a6de18e

• Turn off auto wrap by default

  

  SeanNaren

  committed
  8684f94

  Copy full SHA for 8684f94

Commits on Apr 7, 2021

• Move to trainer.model

  

  SeanNaren

  committed
  76091ae

  Copy full SHA for 76091ae

• fix reference

  

  SeanNaren

  committed
  226d498

  Copy full SHA for 226d498

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  cd63c10

  Copy full SHA for cd63c10

• Remove flag

  

  SeanNaren

  committed
  b881e2f

  Copy full SHA for b881e2f

• Fix imports

  

  SeanNaren

  committed
  e8959be

  Copy full SHA for e8959be

• Fix versions, update docs

  

  SeanNaren

  committed
  52478ac

  Copy full SHA for 52478ac

Commits on Apr 8, 2021

• Fix clip gradients

  

  SeanNaren

  committed
  b7f1896

  Copy full SHA for b7f1896

Commits on Apr 10, 2021

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  a62f8d8

  Copy full SHA for a62f8d8

Commits on Apr 14, 2021

• Merge branch 'master' into feat/fsdp

  

  SeanNaren

  committed
  69c33f1

  Copy full SHA for 69c33f1

• Fixes

  

  SeanNaren

  committed
  9fa26c0

  Copy full SHA for 9fa26c0

• pull

  

  SeanNaren

  committed
  56f23ce

  Copy full SHA for 56f23ce

• Few changes across the board

  

  SeanNaren

  committed
  9ca3f0c

  Copy full SHA for 9ca3f0c

• Fix imports

  

  SeanNaren

  committed
  b53ba36

  Copy full SHA for b53ba36

• Set none

  

  SeanNaren

  committed
  0da5249

  Copy full SHA for 0da5249

• Swap to warnings

  

  SeanNaren

  committed
  90c6479

  Copy full SHA for 90c6479

• Remove fairscale from container

  

  SeanNaren

  committed
  69d8178

  Copy full SHA for 69d8178

• pull

  

  SeanNaren

  committed
  a459d10

  Copy full SHA for a459d10

• Update dockers/base-cuda/Dockerfile

  

  Sean Naren

  authored
  a7842d9

  Copy full SHA for a7842d9

Commits on Apr 15, 2021

• Add defaults, add test to ensure nested wrapper is set correctly

  

  SeanNaren

  committed
  48ee83f

  Copy full SHA for 48ee83f

• Remove deprecation as this will be removed completely

  

  SeanNaren

  committed
  57a696c

  Copy full SHA for 57a696c

Commits on Apr 16, 2021

• Check for nested FSDP wrappers, and omit wrapping algorithm

  

  SeanNaren

  committed
  36889b8

  Copy full SHA for 36889b8

• Merge branch 'master' into feat/fsdp

  Show description for 89b8cb5
  

  SeanNaren

  committed
  89b8cb5

  Copy full SHA for 89b8cb5

Commits on Apr 21, 2021

• Update pytorch_lightning/trainer/connectors/accelerator_connector.py

  Show description for 0c1d2de
  

  Sean Naren

  and

  ananthsub

  authored
  0c1d2de

  Copy full SHA for 0c1d2de

• Address code review points

  

  SeanNaren

  committed
  592bb28

  Copy full SHA for 592bb28

Commits on Apr 26, 2021

• Merge branch 'master' into feat/fsdp

  

  Sean Naren

  authored
  4e230c9

  Copy full SHA for 4e230c9

• Add back missing model that was removed from clipping signature

  

  SeanNaren

  committed
  ca8e586

  Copy full SHA for ca8e586

• Do not pass model through, accelerator does it

  

  SeanNaren

  committed
  54f501d

  Copy full SHA for 54f501d

Commits on Apr 27, 2021

• Merge branch 'master' into feat/fsdp

  

  Sean Naren

  authored
  02925cc

  Copy full SHA for 02925cc

• Fix merge

  

  SeanNaren

  committed
  b67f1a9

  Copy full SHA for b67f1a9

• Fix imports

  

  SeanNaren

  committed
  132eb64

  Copy full SHA for 132eb64

• Changes to precision plugin

  

  SeanNaren

  committed
  e6ce3cf

  Copy full SHA for e6ce3cf

• Require 2 GPU for multi gpu test

  

  SeanNaren

  committed
  01153af

  Copy full SHA for 01153af

Commits on May 2, 2021

• Merge branch 'master' into feat/fsdp

  

  Sean Naren

  authored
  6cfe57d

  Copy full SHA for 6cfe57d

Commits on May 4, 2021

• Use callback in test, swap to DynamicLossScaler from fairscale to test it out

  

  SeanNaren

  committed
  efa81ab

  Copy full SHA for efa81ab

• Disable loss scaler for now

  

  SeanNaren

  committed
  78d52b5

  Copy full SHA for 78d52b5