Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NLP] Improve and unify loading state_dict for community models #7977

Merged
merged 9 commits into from
Jan 19, 2024
Merged

[NLP] Improve and unify loading state_dict for community models #7977

merged 9 commits into from
Jan 19, 2024

Conversation

janekl
Copy link
Collaborator

@janekl janekl commented Dec 6, 2023 •
edited
Loading

What does this PR do ?

  • Provide a common way for loading state_dict for HF models among different conversion scripts
  • Deduplicate tokenizers in the Nemo file produced: the model now comes with a single copy instead of an extra redundant tokenizer file (which is an undesired side effect of load_model function)

Collection: [NLP]

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • Improvement

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

@janekl
Improve and unify loading state_dict for community models
5427d80 
Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
@github-actions github-actions bot added the NLP label Dec 6, 2023
@janekl janekl mentioned this pull request Dec 6, 2023
Closed
8 tasks
@ericharper
Copy link
Collaborator

jenkins

@janekl
Merge branch 'main' into jlasek/unify_load_state_dict_in_conversion
685abc6 
Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
@janekl janekl force-pushed the jlasek/unify_load_state_dict_in_conversion branch from 4570d66 to 685abc6 Compare December 11, 2023 15:20
@janekl janekl mentioned this pull request Dec 27, 2023
Merged
8 tasks
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 4, 2024

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions bot added the stale label Jan 4, 2024
@janekl
Merge branch 'main' into jlasek/unify_load_state_dict_in_conversion
db38b97 
Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
@github-actions github-actions bot removed the stale label Jan 6, 2024
ericharper
ericharper approved these changes Jan 19, 2024
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@ericharper ericharper merged commit e329575 into main Jan 19, 2024
15 checks passed
@ericharper ericharper deleted the jlasek/unify_load_state_dict_in_conversion branch January 19, 2024 16:18
akoumpa pushed a commit to akoumpa/NeMo that referenced this pull request Jan 20, 2024
@janekl @pre-commit-ci @akoumpa
[NLP] Improve and unify loading state_dict for community models (NVID…
5829d98 
…IA#7977)

* Improve and unify loading state_dict for community models

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
stevehuang52 pushed a commit that referenced this pull request Jan 31, 2024
@janekl @pre-commit-ci @stevehuang52
[NLP] Improve and unify loading state_dict for community models (#7977)
f9e1f72 
* Improve and unify loading state_dict for community models

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: stevehuang52 <heh@nvidia.com>
ssh-meister pushed a commit to ssh-meister/NeMo that referenced this pull request Feb 15, 2024
@janekl @pre-commit-ci @sashameister
[NLP] Improve and unify loading state_dict for community models (NVID…
faf79f6 
…IA#7977)

* Improve and unify loading state_dict for community models

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Sasha Meister <ameister@nvidia.com>
pablo-garay pushed a commit that referenced this pull request Mar 19, 2024
@janekl @pre-commit-ci @pablo-garay
[NLP] Improve and unify loading state_dict for community models (#7977)
0fb4f90 
* Improve and unify loading state_dict for community models

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pablo Garay <pagaray@nvidia.com>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
@janekl @pre-commit-ci
[NLP] Improve and unify loading state_dict for community models (NVID…
bf661b6 
…IA#7977)

* Improve and unify loading state_dict for community models

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericharper ericharper ericharper approved these changes

@blahBlahhhJ blahBlahhhJ Awaiting requested review from blahBlahhhJ

Assignees
No one assigned
Labels
NLP
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@janekl @ericharper