Lora ckpt in HF format for NeMo AutoModel #11712

oyilmaz-nvidia · 2024-12-24T23:45:02Z

What does this PR do ?

Adds support to save Lora ckpt in HF format for NeMo automodel.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

examples/llm/peft/hf.py

nemo/lightning/pytorch/strategies/utils.py

akoumpa · 2025-01-06T17:47:55Z

nemo/lightning/pytorch/strategies/utils.py

@@ -117,7 +117,15 @@ def ckpt_to_dir(filepath: Union[str, Path]) -> Path:


 def create_checkpoint_io(wrapping_ckpt_io=None, **kwargs):


can you add a test to make a checkpoint with NeMo and restore it in huggingface? We have tests now for LLM & VLM.

Also, right now checkpoint saving is disabled in the tests, can you turn it on (minor change in the test command)?

Enabled the ckpt savings in those tests. I'll need to address the restore in a separate PR using AutoResume right after this PR.

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

…/NeMo into onur/auto-model-peft-ckpt

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Updating peft test name Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

nemo/lightning/pytorch/strategies/utils.py

+
+        checkpoint_io = HuggingFaceCheckpointIO(lora=kwargs["lora"])
+    else:
+        from nemo.lightning.io.pl import MegatronCheckpointIO


Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

changing the hf vlm test name Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

github-actions · 2025-01-09T18:22:34Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:

************* Module hf
examples/llm/peft/hf.py:28:0: C0301: Line too long (184/119) (line-too-long)
examples/llm/peft/hf.py:24:0: C0116: Missing function or method docstring (missing-function-docstring)
examples/llm/peft/hf.py:63:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.export.vllm_hf_exporter
nemo/export/vllm_hf_exporter.py:62:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/vllm_hf_exporter.py:104:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.io.pl
nemo/lightning/io/pl.py:82:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/io/pl.py:58:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/io/pl.py:64:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:73:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:300:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:305:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.pytorch.strategies.utils
nemo/lightning/pytorch/strategies/utils.py:40:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/pytorch/strategies/utils.py:49:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:57:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:69:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:85:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:120:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:142:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:197:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.73/10

Mitigation guide:

Add sensible and useful docstrings to functions and methods
For trivial methods like getter/setters, consider adding # pylint: disable=C0116 inside the function itself
To disable multiple functions/methods at once, put a # pylint: disable=C0116 before the first and a # pylint: enable=C0116 after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!

github-actions · 2025-01-09T18:22:44Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:

************* Module hf
examples/llm/peft/hf.py:28:0: C0301: Line too long (184/119) (line-too-long)
examples/llm/peft/hf.py:24:0: C0116: Missing function or method docstring (missing-function-docstring)
examples/llm/peft/hf.py:63:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.export.vllm_hf_exporter
nemo/export/vllm_hf_exporter.py:62:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/export/vllm_hf_exporter.py:104:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.io.pl
nemo/lightning/io/pl.py:82:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/io/pl.py:58:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/io/pl.py:64:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:73:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:300:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/io/pl.py:305:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.pytorch.strategies.utils
nemo/lightning/pytorch/strategies/utils.py:40:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/pytorch/strategies/utils.py:49:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:57:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:69:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:85:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:120:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:142:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/pytorch/strategies/utils.py:197:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.73/10

Mitigation guide:

Add sensible and useful docstrings to functions and methods
For trivial methods like getter/setters, consider adding # pylint: disable=C0116 inside the function itself
To disable multiple functions/methods at once, put a # pylint: disable=C0116 before the first and a # pylint: enable=C0116 after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!

github-actions · 2025-01-10T02:08:04Z

[🤖]: Hi @oyilmaz-nvidia 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

akoumpa · 2025-01-10T17:25:46Z

.github/workflows/cicd-main.yml

@@ -3675,7 +3676,7 @@ jobs:
    with:
      RUNNER: self-hosted-azure
      SCRIPT: |
-        TRANSFORMERS_OFFLINE=1 python tests/collections/llm/hf/peft.py --model /home/TestData/nlp/hf_gemma/hf_gemma_2b --max-steps 10 --devices 2 --strategy ddp --disable-ckpt
+        TRANSFORMERS_OFFLINE=1 python tests/collections/llm/hf/peft_hf.py --model /home/TestData/nlp/hf_gemma/hf_gemma_2b --max-steps 10 --devices 2 --strategy ddp --disable-ckpt


@oyilmaz-nvidia thus still has --disable-ckpt

* Save lora ckpt in safetensor and a config Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove hf variable from peft Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * vllm with automodel peft working * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * update examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * removed unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * enable ckpt saving Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> * remove unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fix minor bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

* Save lora ckpt in safetensor and a config Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove hf variable from peft Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * vllm with automodel peft working * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * update examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * removed unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * enable ckpt saving Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> * remove unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fix minor bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Abhinav Garg <abhgarg@nvidia.com>

* Save lora ckpt in safetensor and a config Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * remove hf variable from peft Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * vllm with automodel peft working * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * revert changes Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * update examples Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * removed unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * enable ckpt saving Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> * remove unused import Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> * Apply isort and black reformatting Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> * fix minor bug Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> --------- Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com> Co-authored-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>

oyilmaz-nvidia and others added 9 commits December 23, 2024 13:58

Save lora ckpt in safetensor and a config

eae2eea

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

remove hf variable from peft

99dc538

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

vllm with automodel peft working

70bca02

Apply isort and black reformatting

5540bb1

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

revert changes

7bede6a

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

update examples

035090e

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Fix a few minor issues

11a370c

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Apply isort and black reformatting

d09c0af

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

resolve conflicts

895b284

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

github-advanced-security bot found potential problems Jan 6, 2025

View reviewed changes

examples/llm/peft/hf.py Fixed Show fixed Hide fixed

examples/llm/peft/hf.py Fixed Show fixed Hide fixed

nemo/lightning/pytorch/strategies/utils.py Fixed Show fixed Hide fixed

oyilmaz-nvidia requested review from yaoyu-33 and akoumpa January 6, 2025 13:14

akoumpa reviewed Jan 6, 2025

View reviewed changes

oyilmaz-nvidia and others added 2 commits January 6, 2025 14:29

removed unused import

49679a8

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

enable ckpt saving

15d55e7

Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

github-actions bot added the CI label Jan 6, 2025

oyilmaz-nvidia added the NLP label Jan 6, 2025

oyilmaz-nvidia marked this pull request as ready for review January 6, 2025 22:35

oyilmaz-nvidia requested review from pablo-garay and ko3n1g as code owners January 6, 2025 22:35

ko3n1g previously approved these changes Jan 7, 2025

View reviewed changes

remove unused import

6f68721

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

oyilmaz-nvidia dismissed ko3n1g’s stale review via 6f68721 January 7, 2025 17:16

github-actions bot removed the NLP label Jan 7, 2025

Apply isort and black reformatting

17afc4d

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

oyilmaz-nvidia added skip-docs Run CICD labels Jan 7, 2025

Merge branch 'main' into onur/auto-model-peft-ckpt

35609fd

oyilmaz-nvidia removed Run CICD skip-docs labels Jan 7, 2025

oyilmaz-nvidia enabled auto-merge (squash) January 8, 2025 08:38

Merge branch 'main' into onur/auto-model-peft-ckpt

dd0bce1

oyilmaz-nvidia added Run CICD and removed Run CICD labels Jan 8, 2025

oyilmaz-nvidia and others added 5 commits January 8, 2025 11:50

fix circular import issue

ff3eb44

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Merge branch 'onur/auto-model-peft-ckpt' of https://github.com/NVIDIA…

d0d9b74

…/NeMo into onur/auto-model-peft-ckpt

Apply isort and black reformatting

f57283d

Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>

Update cicd-main.yml

b0bfc67

Updating peft test name Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

Merge branch 'main' into onur/auto-model-peft-ckpt

da69286

oyilmaz-nvidia added Run CICD and removed Run CICD labels Jan 8, 2025

github-advanced-security bot found potential problems Jan 8, 2025

View reviewed changes

nemo/lightning/pytorch/strategies/utils.py

checkpoint_io = HuggingFaceCheckpointIO(lora=kwargs["lora"])

else:

from nemo.lightning.io.pl import MegatronCheckpointIO

Check notice

Code scanning / CodeQL

Cyclic import Note

Import of module

nemo.lightning.io.pl

Loading
begins an import cycle.

fixing the datasource

a18adab

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

oyilmaz-nvidia added Run CICD and removed Run CICD labels Jan 9, 2025

oyilmaz-nvidia and others added 3 commits January 9, 2025 10:15

renaming to avoid import error

29ee000

Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>

Update cicd-main.yml

dbd4e01

changing the hf vlm test name Signed-off-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>

Merge branch 'main' into onur/auto-model-peft-ckpt

8a47a7f

oyilmaz-nvidia added Run CICD and removed Run CICD labels Jan 9, 2025

pablo-garay approved these changes Jan 10, 2025

View reviewed changes

oyilmaz-nvidia merged commit 9799051 into main Jan 10, 2025
201 of 204 checks passed

oyilmaz-nvidia deleted the onur/auto-model-peft-ckpt branch January 10, 2025 16:51

akoumpa reviewed Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lora ckpt in HF format for NeMo AutoModel #11712

Lora ckpt in HF format for NeMo AutoModel #11712

oyilmaz-nvidia commented Dec 24, 2024

akoumpa Jan 6, 2025

oyilmaz-nvidia Jan 8, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 10, 2025

akoumpa Jan 10, 2025

		@@ -117,7 +117,15 @@ def ckpt_to_dir(filepath: Union[str, Path]) -> Path:


		def create_checkpoint_io(wrapping_ckpt_io=None, **kwargs):

Lora ckpt in HF format for NeMo AutoModel #11712

Lora ckpt in HF format for NeMo AutoModel #11712

Conversation

oyilmaz-nvidia commented Dec 24, 2024

What does this PR do ?

akoumpa Jan 6, 2025

Choose a reason for hiding this comment

oyilmaz-nvidia Jan 8, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 10, 2025

akoumpa Jan 10, 2025

Choose a reason for hiding this comment