Code

Add FMS datasets#1

Code

Merged

daviswer merged 26 commits intomaindaviswer/torchtitan:mainfrom fms-datasetsdaviswer/torchtitan:fms-datasetsCopy head branch name to clipboardMay 31, 2024

+1,350-11

Commits

Commits on May 21, 2024

• Add datasets, dataloader, swap out, update cfg

  daviswer

  committed
  db20ea4

  Copy full SHA for db20ea4

• Actually add dataset file

  daviswer

  committed
  9dbe4be

  Copy full SHA for 9dbe4be

• Update llama3_8b.toml

  lchu6

  authored
  79cbe06

  Copy full SHA for 79cbe06

• Update llama3_8b.toml

  lchu6

  authored
  18f6c0d

  Copy full SHA for 18f6c0d

• Update llama3_8b.toml

  lchu6

  authored
  9c94d3a

  Copy full SHA for 9c94d3a

• Update fms_datasets.py

  lchu6

  authored
  465e972

  Copy full SHA for 465e972

Commits on May 22, 2024

• Cast inputs/targs to long

  daviswer

  committed
  2fd356a

  Copy full SHA for 2fd356a

• Update llama3_8b.toml

  lchu6

  authored
  b967c58

  Copy full SHA for b967c58

• Update data ckp to full path

  daviswer

  committed
  5baa18e

  Copy full SHA for 5baa18e

• Merge branch 'fms-datasets' of github.com:daviswer/torchtitan into fms-datasets

  daviswer

  committed
  ca8503f

  Copy full SHA for ca8503f

• Update llama3_8b.toml

  lchu6

  authored
  48e2cd1

  Copy full SHA for 48e2cd1

• Update llama3_8b.toml

  lchu6

  authored
  db79072

  Copy full SHA for db79072

Commits on May 23, 2024

• Reconcile ckp naming schemes

  daviswer

  committed
  2597c05

  Copy full SHA for 2597c05

• Swap ckp folder names from step_ to step-

  daviswer

  committed
  2bbed13

  Copy full SHA for 2bbed13

Commits on May 29, 2024

• add llama3-tokenized c4 mini

  lchu6

  committed
  ad201e7

  Copy full SHA for ad201e7

• Update llama3_8b.toml

  lchu6

  authored
  5784c76

  Copy full SHA for 5784c76

• Update llama3_8b.toml

  lchu6

  authored
  5664b8c

  Copy full SHA for 5664b8c

• add llama2 tokenized data

  lchu6

  committed
  66e7b2b

  Copy full SHA for 66e7b2b

• Merge branch 'main' into fms-datasets

  daviswer

  authored
  dbf26e0

  Copy full SHA for dbf26e0

• Make dataloader usage flaggable, remove refs to fms

  daviswer

  committed
  de705af

  Copy full SHA for de705af

• Add back old datapath flag, separate out new flags in cfg

  daviswer

  committed
  1f1fdc9

  Copy full SHA for 1f1fdc9

Commits on May 30, 2024

• up

  lchu6

  committed
  e79ec4f

  Copy full SHA for e79ec4f

• up

  lchu6

  committed
  145ab47

  Copy full SHA for 145ab47

• up

  lchu6

  committed
  c4e6b0b

  Copy full SHA for c4e6b0b

• up

  lchu6

  committed
  b0eae7d

  Copy full SHA for b0eae7d

• up

  lchu6

  committed
  d45ed13

  Copy full SHA for d45ed13