Virtual pipeline parallel support for LoRA in NLPAdapterModelMixin #11128
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eline parallel + LoRA Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
Signed-off-by: vysarge <vysarge@users.noreply.github.com>
erhoo82
reviewed
Nov 1, 2024
Comment on lines
+468
to
+469
| and "model_0" in state_dict | ||
| and len(state_dict["model_0"]) == 0 |
There was a problem hiding this comment.
Can you explain the logic of this code by comment?
Signed-off-by: Valerie Sarge <vsarge@nvidia.com>
erhoo82
approved these changes
Nov 1, 2024
ko3n1g
pushed a commit
that referenced
this pull request
Nov 1, 2024
…11128) * Update NLPAdapterModelMixin to handle model structure for virtual pipeline parallel + LoRA Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up assert guard Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up ValueError raise Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Apply isort and black reformatting Signed-off-by: vysarge <vysarge@users.noreply.github.com> * documentation Signed-off-by: Valerie Sarge <vsarge@nvidia.com> --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Signed-off-by: vysarge <vysarge@users.noreply.github.com> Co-authored-by: vysarge <vysarge@users.noreply.github.com>
vysarge
added a commit
that referenced
this pull request
Nov 1, 2024
…11128) (#11135) * Update NLPAdapterModelMixin to handle model structure for virtual pipeline parallel + LoRA * Clean up assert guard * Clean up ValueError raise * Apply isort and black reformatting * documentation --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Signed-off-by: vysarge <vysarge@users.noreply.github.com> Co-authored-by: Valerie Sarge <vsarge@nvidia.com> Co-authored-by: vysarge <vysarge@users.noreply.github.com>
hainan-xv
pushed a commit
to hainan-xv/NeMo
that referenced
this pull request
Nov 5, 2024
…VIDIA#11128) * Update NLPAdapterModelMixin to handle model structure for virtual pipeline parallel + LoRA Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up assert guard Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up ValueError raise Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Apply isort and black reformatting Signed-off-by: vysarge <vysarge@users.noreply.github.com> * documentation Signed-off-by: Valerie Sarge <vsarge@nvidia.com> --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Signed-off-by: vysarge <vysarge@users.noreply.github.com> Co-authored-by: vysarge <vysarge@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>
lilyw97
pushed a commit
to lilyw97/NeMo
that referenced
this pull request
Nov 13, 2024
…VIDIA#11128) * Update NLPAdapterModelMixin to handle model structure for virtual pipeline parallel + LoRA Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up assert guard Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up ValueError raise Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Apply isort and black reformatting Signed-off-by: vysarge <vysarge@users.noreply.github.com> * documentation Signed-off-by: Valerie Sarge <vsarge@nvidia.com> --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Signed-off-by: vysarge <vysarge@users.noreply.github.com> Co-authored-by: vysarge <vysarge@users.noreply.github.com>
HuiyingLi
pushed a commit
to HuiyingLi/NeMo
that referenced
this pull request
Nov 15, 2024
…VIDIA#11128) * Update NLPAdapterModelMixin to handle model structure for virtual pipeline parallel + LoRA Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up assert guard Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Clean up ValueError raise Signed-off-by: Valerie Sarge <vsarge@nvidia.com> * Apply isort and black reformatting Signed-off-by: vysarge <vysarge@users.noreply.github.com> * documentation Signed-off-by: Valerie Sarge <vsarge@nvidia.com> --------- Signed-off-by: Valerie Sarge <vsarge@nvidia.com> Signed-off-by: vysarge <vysarge@users.noreply.github.com> Co-authored-by: vysarge <vysarge@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Corrects handling of
model.virtual_pipeline_model_parallel_size> 1 alongsidemodel.peft.peft_scheme=lora; before this fix adapters are attached only to the first virtual pipeline chunk.Collection: nlp
Changelog
parallel_state.is_pipeline_first_stage()to account for virtual pipeline stage_get_all_keys,_check_and_add_peft_cfg,setup_optimizer_param_groups,set_tunable_base_params,tie_weights,get_peft_state_dict,load_state_dict) to apply LoRA changes to all layers rather than only those in the first chunk and to write and read checkpoints correctly when virtual pipeline parallel is usedUsage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information