Code

Enable huggingface tokenizer#1268

Code

Open

msiddaiah wants to merge 6 commits intoNVIDIA:mainNVIDIA/Megatron-LM:mainfrom ROCm:enable_huggingface_tokenizerROCm/Megatron-LM:enable_huggingface_tokenizerCopy head branch name to clipboard

+252-52

Commits

Commits on Oct 10, 2024

• add jenkins pipeline

  gurpreet-dhami

  committed
  717e7cd

  Copy full SHA for 717e7cd

Commits on Oct 11, 2024

• fix runtime error for rocm (#3)

  gurpreet-dhami

  authored
  0c4709e

  Copy full SHA for 0c4709e

• Merge remote-tracking branch 'upstream/main' into upstream_sync

  gurpreet-dhami

  committed
  e8c077c

  Copy full SHA for e8c077c

Commits on Oct 18, 2024

• Merge pull request #6 from ROCm/upstream_sync

  Show description for 42b34ba
  gurpreet-dhami

  authored
  42b34ba

  Copy full SHA for 42b34ba

Commits on Oct 22, 2024

• rope enable through TE (#4)

  Show description for 2b855ba
  gurpreet-dhami

  authored
  2b855ba

  Copy full SHA for 2b855ba

Commits on Oct 30, 2024

• Enable HuggingFaceTokenizer in preprocessing

  

  Manjunath Siddaiah

  committed
  69c0c06

  Copy full SHA for 69c0c06