Add support and recipes for HF models via AutoModelForCausalLM #10962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akoumpa
force-pushed
the
akoumparouli/hf_lit_module
branch
2 times, most recently
from
5a01f6c to
887ed5a
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
akoumpa
force-pushed
the
akoumparouli/hf_lit_module
branch
14 times, most recently
from
723f74c to
2276451
Compare
akoumpa
force-pushed
the
akoumparouli/hf_lit_module
branch
7 times, most recently
from
f431889 to
22931e2
Compare
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
akoumpa
force-pushed
the
akoumparouli/hf_lit_module
branch
from
2733fda to
8d1d6aa
Compare
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
akoumpa
force-pushed
the
akoumparouli/hf_lit_module
branch
from
c040e20 to
bf8ee45
Compare
Signed-off-by: akoumpa <akoumpa@users.noreply.github.com>
hemildesai
approved these changes
Oct 22, 2024
|
[🤖]: Hi @akoumpa 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully So it might be time to merge this PR or get some approvals I'm just a bot so I'll leave it you what to do next. //cc @pablo-garay @ko3n1g |
akoumpa
added a commit
that referenced
this pull request
Oct 24, 2024
* initial hf_lit_module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make sft gpt dataset sanity check optional Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HF sft example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Rename HfLitModule to HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update default model id Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rank&world_size as params Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix mbs in example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix for fsdp and logger Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make loss_fn configurable Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove optim from HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add pytorch native optim Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add hfAutoModel pretrain nemorun recipe Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove debug Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * tokenizer fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename pytorch_adam to pytorch_adam_with_cosine_annealing Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix no_weight_decay_cond Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * switch to flat_lr optim for example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove imports & update docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add a tokenizer setter to allow it to work with nemo/collections/llm/api.py::_use_tokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * allow loss_mask to be none Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add HF-dataset lightning module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * check if pad_token_id is None Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename hf_lit_module.py to hf_auto_model.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * class rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HfAutoModelForCausalLM Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add option to start with random weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add check in megatron-strategy Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * drop mcore sampler from squadmodule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make megatron_sampler optional in HfDatasetDataModule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * copyright Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use is_hf_model to mark hf classes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
yashaswikarnati
pushed a commit
that referenced
this pull request
Oct 24, 2024
* initial hf_lit_module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make sft gpt dataset sanity check optional Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HF sft example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Rename HfLitModule to HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update default model id Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rank&world_size as params Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix mbs in example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix for fsdp and logger Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make loss_fn configurable Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove optim from HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add pytorch native optim Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add hfAutoModel pretrain nemorun recipe Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove debug Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * tokenizer fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename pytorch_adam to pytorch_adam_with_cosine_annealing Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix no_weight_decay_cond Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * switch to flat_lr optim for example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove imports & update docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add a tokenizer setter to allow it to work with nemo/collections/llm/api.py::_use_tokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * allow loss_mask to be none Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add HF-dataset lightning module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * check if pad_token_id is None Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename hf_lit_module.py to hf_auto_model.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * class rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HfAutoModelForCausalLM Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add option to start with random weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add check in megatron-strategy Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * drop mcore sampler from squadmodule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make megatron_sampler optional in HfDatasetDataModule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * copyright Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use is_hf_model to mark hf classes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com>
titu1994
pushed a commit
that referenced
this pull request
Oct 28, 2024
* initial hf_lit_module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make sft gpt dataset sanity check optional Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HF sft example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Rename HfLitModule to HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update default model id Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rank&world_size as params Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix mbs in example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix for fsdp and logger Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make loss_fn configurable Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove optim from HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add pytorch native optim Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add hfAutoModel pretrain nemorun recipe Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove debug Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * tokenizer fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename pytorch_adam to pytorch_adam_with_cosine_annealing Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix no_weight_decay_cond Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * switch to flat_lr optim for example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove imports & update docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add a tokenizer setter to allow it to work with nemo/collections/llm/api.py::_use_tokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * allow loss_mask to be none Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add HF-dataset lightning module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * check if pad_token_id is None Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename hf_lit_module.py to hf_auto_model.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * class rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HfAutoModelForCausalLM Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add option to start with random weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add check in megatron-strategy Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * drop mcore sampler from squadmodule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make megatron_sampler optional in HfDatasetDataModule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * copyright Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use is_hf_model to mark hf classes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com>
hainan-xv
pushed a commit
to hainan-xv/NeMo
that referenced
this pull request
Nov 5, 2024
…A#10962) * initial hf_lit_module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make sft gpt dataset sanity check optional Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HF sft example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Rename HfLitModule to HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update default model id Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rank&world_size as params Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix mbs in example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix for fsdp and logger Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make loss_fn configurable Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove optim from HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add pytorch native optim Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add hfAutoModel pretrain nemorun recipe Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove debug Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * tokenizer fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename pytorch_adam to pytorch_adam_with_cosine_annealing Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix no_weight_decay_cond Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * switch to flat_lr optim for example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove imports & update docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add a tokenizer setter to allow it to work with nemo/collections/llm/api.py::_use_tokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * allow loss_mask to be none Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add HF-dataset lightning module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * check if pad_token_id is None Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename hf_lit_module.py to hf_auto_model.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * class rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HfAutoModelForCausalLM Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add option to start with random weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add check in megatron-strategy Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * drop mcore sampler from squadmodule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make megatron_sampler optional in HfDatasetDataModule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * copyright Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use is_hf_model to mark hf classes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
HuiyingLi
pushed a commit
to HuiyingLi/NeMo
that referenced
this pull request
Nov 15, 2024
…A#10962) * initial hf_lit_module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make sft gpt dataset sanity check optional Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HF sft example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Rename HfLitModule to HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update default model id Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * move rank&world_size as params Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix mbs in example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix for fsdp and logger Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make loss_fn configurable Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove optim from HfAutoModel Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add pytorch native optim Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add hfAutoModel pretrain nemorun recipe Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove debug Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale imports Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * tokenizer fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename pytorch_adam to pytorch_adam_with_cosine_annealing Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * small refactor Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix no_weight_decay_cond Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fix Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * switch to flat_lr optim for example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> * remove imports & update docstrings Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add a tokenizer setter to allow it to work with nemo/collections/llm/api.py::_use_tokenizer Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * remove unused import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * allow loss_mask to be none Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add HF-dataset lightning module Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * check if pad_token_id is None Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename hf_lit_module.py to hf_auto_model.py Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * class rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * update example Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * HfAutoModelForCausalLM Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rm stale import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add option to start with random weights Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * add check in megatron-strategy Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * rename param Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * drop mcore sampler from squadmodule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * make megatron_sampler optional in HfDatasetDataModule Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * copyright Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * use is_hf_model to mark hf classes Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Apply isort and black reformatting Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: akoumpa <akoumpa@users.noreply.github.com> Co-authored-by: akoumpa <akoumpa@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Can be invoked with
nemoor via the examplethe example:
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information