Mixtral to NeMo conversion script. #8155

akoumpa · 2024-01-11T17:54:53Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

scripts/nlp_language_modeling/convert_hf_mixtral_to_nemo.py

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

cdj0311 · 2024-01-12T07:59:04Z

Could you provide the convert NeMo to huggingface script?

akoumpa · 2024-01-12T10:40:02Z

@cdj0311 will try to have the NeMo to HF by next week

scripts/nlp_language_modeling/convert_nemo_mixtral_to_hf.py

ericharper · 2024-01-19T16:24:57Z

jenkins

ericharper · 2024-01-20T01:59:47Z

jenkins

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

ericharper · 2024-01-25T18:47:20Z

jenkins

ericharper · 2024-01-25T18:49:11Z

jenkins

…cify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa · 2024-01-26T20:02:00Z

jenkins

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa · 2024-01-26T20:29:25Z

jenkins

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa · 2024-01-26T22:02:27Z

jenkins

for more information, see https://pre-commit.ci

akoumpa · 2024-01-26T22:17:26Z

jenkins

ericharper

LGTM. Thanks!

* HF-Mixtral to NeMo conversion script. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Pass MoE options from NeMo config to TransformerConfig. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add version check for get_gpt_layer_with_transformer_engine_spec Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Determine MoE support by attempting to import MoETokenDispatcher. Using importlib.metadata.version would be an alternative, however, a) it requires having mcore installed via pip (not always the case) and b) one might override megatron's location (e.g. via PYTHONPATH) and as a result would get inaccurate version from importlib.metadata. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-NeMo to Mixtral-HF converter. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fixup: Update mcore_supports_moe due to file rename in upcoming MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-converters: use `set_expert_model_parallel_world_size` to specify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Jenkins: install lightning. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Match latest MoE parameter names. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com>

* HF-Mixtral to NeMo conversion script. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Pass MoE options from NeMo config to TransformerConfig. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add version check for get_gpt_layer_with_transformer_engine_spec Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Determine MoE support by attempting to import MoETokenDispatcher. Using importlib.metadata.version would be an alternative, however, a) it requires having mcore installed via pip (not always the case) and b) one might override megatron's location (e.g. via PYTHONPATH) and as a result would get inaccurate version from importlib.metadata. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-NeMo to Mixtral-HF converter. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fixup: Update mcore_supports_moe due to file rename in upcoming MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-converters: use `set_expert_model_parallel_world_size` to specify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Jenkins: install lightning. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Match latest MoE parameter names. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: stevehuang52 <heh@nvidia.com>

janekl · 2024-02-12T12:25:52Z

Hi @ericharper. I browsed CodeQL suggestions in this pull request. This is a good example when Flake8 is better than CodeQL: CodeQL fails to detect undefined variables (a simple issue), see head_num here, while Flake8 catches them. I strongly suggest to start using Flake8 complementary to CodeQL in CI.

janekl · 2024-02-12T12:31:18Z

Another example are the unused imports from megatron.core.utils import init_method_normal, scaled_init_method_normal here. This file is frequently edited btw. Possibly CodeQL misses this due to import guards, but I'm not sure. cc @ericharper

* HF-Mixtral to NeMo conversion script. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Pass MoE options from NeMo config to TransformerConfig. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add version check for get_gpt_layer_with_transformer_engine_spec Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Determine MoE support by attempting to import MoETokenDispatcher. Using importlib.metadata.version would be an alternative, however, a) it requires having mcore installed via pip (not always the case) and b) one might override megatron's location (e.g. via PYTHONPATH) and as a result would get inaccurate version from importlib.metadata. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-NeMo to Mixtral-HF converter. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fixup: Update mcore_supports_moe due to file rename in upcoming MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-converters: use `set_expert_model_parallel_world_size` to specify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Jenkins: install lightning. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Match latest MoE parameter names. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Sasha Meister <ameister@nvidia.com>

* HF-Mixtral to NeMo conversion script. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Pass MoE options from NeMo config to TransformerConfig. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add version check for get_gpt_layer_with_transformer_engine_spec Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Determine MoE support by attempting to import MoETokenDispatcher. Using importlib.metadata.version would be an alternative, however, a) it requires having mcore installed via pip (not always the case) and b) one might override megatron's location (e.g. via PYTHONPATH) and as a result would get inaccurate version from importlib.metadata. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-NeMo to Mixtral-HF converter. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fixup: Update mcore_supports_moe due to file rename in upcoming MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-converters: use `set_expert_model_parallel_world_size` to specify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Jenkins: install lightning. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Match latest MoE parameter names. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com>

* HF-Mixtral to NeMo conversion script. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Pass MoE options from NeMo config to TransformerConfig. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Add version check for get_gpt_layer_with_transformer_engine_spec Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Determine MoE support by attempting to import MoETokenDispatcher. Using importlib.metadata.version would be an alternative, however, a) it requires having mcore installed via pip (not always the case) and b) one might override megatron's location (e.g. via PYTHONPATH) and as a result would get inaccurate version from importlib.metadata. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-NeMo to Mixtral-HF converter. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * fixup: Update mcore_supports_moe due to file rename in upcoming MoE Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Mixtral-converters: use `set_expert_model_parallel_world_size` to specify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Fix import Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Jenkins: install lightning. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * Match latest MoE parameter names. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com>

github-actions bot added the NLP label Jan 11, 2024

akoumpa commented Jan 11, 2024

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Jan 11, 2024

View reviewed changes

scripts/nlp_language_modeling/convert_hf_mixtral_to_nemo.py Dismissed Show dismissed Hide dismissed

scripts/nlp_language_modeling/convert_hf_mixtral_to_nemo.py Dismissed Show dismissed Hide dismissed

scripts/nlp_language_modeling/convert_hf_mixtral_to_nemo.py Fixed Show fixed Hide fixed

akoumpa force-pushed the mistral_moe_support/akoumparouli branch 2 times, most recently from 85a4249 to eccdece Compare January 11, 2024 21:11

github-advanced-security bot found potential problems Jan 11, 2024

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Fixed Show fixed Hide fixed

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from eccdece to e32c3f2 Compare January 17, 2024 00:22

github-advanced-security bot found potential problems Jan 17, 2024

View reviewed changes

akoumpa force-pushed the mistral_moe_support/akoumparouli branch 3 times, most recently from 10804f8 to e22b72e Compare January 17, 2024 21:40

akoumpa marked this pull request as ready for review January 18, 2024 17:58

akoumpa requested a review from ericharper January 18, 2024 17:58

akoumpa force-pushed the mistral_moe_support/akoumparouli branch 5 times, most recently from 65c3935 to 3f985b6 Compare January 19, 2024 09:54

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from 4c45018 to a5ba328 Compare January 20, 2024 00:27

github-actions bot added core Changes to NeMo Core CI common labels Jan 20, 2024

github-actions bot removed core Changes to NeMo Core CI common labels Jan 22, 2024

HF-Mixtral to NeMo conversion script.

e4708e6

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from d70c027 to d926b1a Compare January 22, 2024 19:29

akoumpa added 2 commits January 25, 2024 11:51

Mixtral-converters: use set_expert_model_parallel_world_size to spe…

3b22247

…cify MoE world size. Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

Merge branch 'main' into mistral_moe_support/akoumparouli

171f37f

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from aa09f9e to 171f37f Compare January 25, 2024 19:55

akoumpa and others added 2 commits January 26, 2024 10:42

Fix import

404917c

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

02a2a2f

for more information, see https://pre-commit.ci Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from b3af500 to 02a2a2f Compare January 26, 2024 18:44

Merge branch 'main' into mistral_moe_support/akoumparouli

3640121

Jenkins: install lightning.

ae5bff6

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

github-actions bot added the CI label Jan 26, 2024

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from 33a71fd to 2ee101b Compare January 26, 2024 21:58

Match latest MoE parameter names.

efdd42e

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

akoumpa force-pushed the mistral_moe_support/akoumparouli branch from f3c8be2 to efdd42e Compare January 26, 2024 22:01

pre-commit-ci bot and others added 2 commits January 26, 2024 22:02

[pre-commit.ci] auto fixes from pre-commit.com hooks

7ae8f12

for more information, see https://pre-commit.ci

Merge branch 'main' into mistral_moe_support/akoumparouli

5dc3283

ericharper approved these changes Jan 27, 2024

View reviewed changes

ericharper merged commit 13c1db4 into NVIDIA:main Jan 27, 2024
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mixtral to NeMo conversion script. #8155

Mixtral to NeMo conversion script. #8155

akoumpa commented Jan 11, 2024 •

edited

Loading

cdj0311 commented Jan 12, 2024

akoumpa commented Jan 12, 2024

ericharper commented Jan 19, 2024

ericharper commented Jan 20, 2024

ericharper commented Jan 25, 2024

ericharper commented Jan 25, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

ericharper left a comment

janekl commented Feb 12, 2024 •

edited

Loading

janekl commented Feb 12, 2024 •

edited

Loading

Mixtral to NeMo conversion script. #8155

Mixtral to NeMo conversion script. #8155

Conversation

akoumpa commented Jan 11, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

cdj0311 commented Jan 12, 2024

akoumpa commented Jan 12, 2024

ericharper commented Jan 19, 2024

ericharper commented Jan 20, 2024

ericharper commented Jan 25, 2024

ericharper commented Jan 25, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

akoumpa commented Jan 26, 2024

ericharper left a comment

Choose a reason for hiding this comment

janekl commented Feb 12, 2024 • edited Loading

janekl commented Feb 12, 2024 • edited Loading

akoumpa commented Jan 11, 2024 •

edited

Loading

janekl commented Feb 12, 2024 •

edited

Loading

janekl commented Feb 12, 2024 •

edited

Loading