Code

fix optimizer reset for relora sft#1414

Code

Merged

winglian merged 5 commits intomainaxolotl-ai-cloud/axolotl:mainfrom relora-fix-optimizer-resetaxolotl-ai-cloud/axolotl:relora-fix-optimizer-resetCopy head branch name to clipboardDec 3, 2024

+64-30

Commits

Commits on Dec 3, 2024

• fix optimizer reset

  winglian

  committed
  d9b6ce4

  Copy full SHA for d9b6ce4

• set states to reset for 8bit optimizers and handle quantile runtime error for embeddings

  winglian

  committed
  9f57eae

  Copy full SHA for 9f57eae

• fix relora test to check grad_norm

  winglian

  committed
  f4cb178

  Copy full SHA for f4cb178

• use flash attn for relora and tweak hyperparams for test

  winglian

  committed
  4d8a839

  Copy full SHA for 4d8a839

• fix messages field for test dataset

  winglian

  committed
  13b5865

  Copy full SHA for 13b5865