Code

Prompt Learning Pipeline Parallel#4291

Code

Merged

vadam5 merged 50 commits intomainNVIDIA/NeMo:mainfrom prompt-learning-pipeline-parallelNVIDIA/NeMo:prompt-learning-pipeline-parallelCopy head branch name to clipboardJun 9, 2022

+332-192

Commits

Commits on May 13, 2022

• Added get_forward_output_and_loss_func and updated train/val steps

  Show description for cbd6f44
  vadam5

  committed
  cbd6f44

Commits on May 17, 2022

• Added preprocess flag before prompt table/encoder access

  Show description for 87933f8
  vadam5

  committed
  87933f8

Commits on May 31, 2022

• Resolved merge conflicts

  Show description for e05333d
  vadam5

  committed
  e05333d

• Made two optimizer groups, one for frozen, one for soft prompt

  Show description for 9a8c7bd
  vadam5

  committed
  9a8c7bd

Commits on Jun 1, 2022

• Pipeline parallel working

  Show description for b13448d
  vadam5

  committed
  b13448d

• Still figuring out setting lr/sched for one param group

  Show description for 4f17803
  vadam5

  committed
  4f17803

Commits on Jun 2, 2022

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  committed
  1790ea0

Commits on Jun 5, 2022

• Set betas to zero

  Show description for 65eb4f6
  vadam5

  committed
  65eb4f6

Commits on Jun 6, 2022

• Only unfreeze one sublayer with lr 0.0

  Show description for 6ae7bc7
  vadam5

  committed
  6ae7bc7

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  604a49b

Commits on Jun 7, 2022

• Pipeline parallel working w/ one optimizer

  Show description for fc21396
  vadam5

  committed
  fc21396

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  8a4e4bb

• Trying to fix Jenkins file

  Show description for 78ee9ed
  vadam5

  committed
  78ee9ed

• Trying to fix Jenkins file

  Show description for c9755ed
  vadam5

  committed
  c9755ed

• Getting updated jenkins test to work

  Show description for 55faa1e
  vadam5

  committed
  55faa1e

• Getting updated jenkins test to work

  Show description for 22e52e2
  vadam5

  committed
  22e52e2

• added prompt learning tp and pp CI tests

  Show description for f230117
  vadam5

  committed
  f230117

• Added amp_o2 model compatibility

  Show description for 9ef0f4a
  vadam5

  committed
  9ef0f4a

• Made CI test smaller

  Show description for 5bde977
  vadam5

  committed
  5bde977

• Still trying to get Jenkins to work

  Show description for 7a0de7f
  vadam5

  committed
  7a0de7f

• Still trying to get Jenkins to work

  Show description for 181de71
  vadam5

  committed
  181de71

• Temporarily moving prompt learning CI test to beginning

  Show description for 5a863cd
  vadam5

  committed
  5a863cd

• Changing the layer being unfrozen

  Show description for fe861ce
  vadam5

  committed
  fe861ce

• debug jenkins

  Show description for 706e18a
  vadam5

  committed
  706e18a

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  5624820

• Move pp unfreeze to init

  Show description for 5133377
  vadam5

  committed
  5133377

• Try to make Jenkins test parallel

  Show description for e58a4a7
  vadam5

  committed
  e58a4a7

• Fix python formatting

  Show description for 23c5a08
  vadam5

  committed
  23c5a08

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  f3fd379

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  911fac8

• Moved prompt learning tests back to where they belong

  Show description for 49ff80b
  vadam5

  committed
  49ff80b

• Merge branch 'prompt-learning-pipeline-parallel' of https://github.com/NVIDIA/NeMo into prompt-learning-pipeline-parallel

  vadam5

  committed
  3c98b33

• add back checkpoint convertion CI test

  Show description for 61e2ffc
  vadam5

  committed
  61e2ffc

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  7c6b8c8

Commits on Jun 8, 2022

• Revert "add back checkpoint convertion CI test"

  Show description for d5dd52c
  vadam5

  committed
  d5dd52c

• Add back checkpoint conversion test

  Show description for 8ff61ca
  vadam5

  committed
  8ff61ca

• Setting requires grad to True everywhere

  Show description for 388d10d
  vadam5

  committed
  388d10d

• Merge branch 'main' into prompt-learning-pipeline-parallel

  ericharper

  authored
  0949c3d

• Updated config comments and simplified param group code

  Show description for 68e3384
  vadam5

  committed
  68e3384

• Merge branch 'prompt-learning-pipeline-parallel' of https://github.com/NVIDIA/NeMo into prompt-learning-pipeline-parallel

  vadam5

  committed
  ac0122f

• Added comment on frozen_model having lr=0.0

  Show description for f127c35
  vadam5

  committed
  f127c35

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  bca3c2a

• Merge branch 'main' into prompt-learning-pipeline-parallel

  ericharper

  authored
  59043e2

• Added configure optimizers methods

  Show description for bcb318c
  vadam5

  committed
  bcb318c

• Merge branch 'prompt-learning-pipeline-parallel' of https://github.com/NVIDIA/NeMo into prompt-learning-pipeline-parallel

  vadam5

  committed
  1986b59

• Set amp_o2 to false

  Show description for 54f60c3
  vadam5

  committed
  54f60c3

Commits on Jun 9, 2022

• removed o2 code

  Show description for c120e6a
  vadam5

  committed
  c120e6a

• Python formatting fix

  Show description for 936bffd
  vadam5

  committed
  936bffd

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  6aaf222

• Merge branch 'main' into prompt-learning-pipeline-parallel

  vadam5

  authored
  68b2fa3