Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug Transformer Engine FP8 support with Megatron-core infrastructure #6740

Merged
merged 4 commits into from
Jun 1, 2023

Conversation

timmoon10
Copy link
Collaborator

What does this PR do ?

Debugs Transformer Engine FP8 support with Megatron-core infrastructure

Collection: NLP

Changelog

  • Construct FP8 amax reduction group

Usage

Set transformer_engine: True and fp8: True in the config.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Pinging @aklife97 @yen-shi @erhoo82 @ksivaman

Additional Information

timmoon10 and others added 2 commits May 26, 2023 14:37
Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
timmoon10 and others added 2 commits May 31, 2023 14:38
@github-actions github-actions bot removed the CI label Jun 1, 2023
@aklife97 aklife97 merged commit 4bbb3c6 into r1.19.0 Jun 1, 2023
@aklife97 aklife97 deleted the tmoon/fp8-amax-group-debug branch June 1, 2023 19:07
github-actions bot pushed a commit that referenced this pull request Jun 1, 2023
#6740)

* Construct FP8 amax reduction group

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* update core for CI

Signed-off-by: Abhinav Khattar <aklife97@gmail.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Abhinav Khattar <aklife97@gmail.com>
Co-authored-by: Abhinav Khattar <aklife97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants