Automatic mixed precision #2359
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benson31
reviewed
Oct 26, 2023
benson31
reviewed
Oct 27, 2023
benson31
reviewed
Oct 27, 2023
benson31
approved these changes
Oct 27, 2023
benson31
approved these changes
Oct 29, 2023
There was a problem hiding this comment.
Still looking good after the rebase/recent commits. I touched the memory_profile callback and added an example of how to use the SwitchDispatcher. I found it a bit clunkier than I remembered to use, but I think I have some ideas to improve it in future H2 versions (in particular, I want to make it possible to just use a lambda as the functor).
Also, please clang-format the whole PR.
ndryden
force-pushed
the
amp
branch
2 times, most recently
from
dc5c648 to
b5b91c8
Compare
benson31
approved these changes
Nov 2, 2023
benson31
reviewed
Nov 6, 2023
This allows specifying float as a datatype even when it would not be the default.
- Permit using tensor cores by default, if no data conversion is needed. - Change "no tensor ops" mode to prevent TF32 conversion. - Add mode to allow conversion.
This is a hack that we should revisit.
This only matters on cuDNN, where we want "pseudo half".
Co-authored-by: Tom Benson <benson31@llnl.gov>
Co-authored-by: Tom Benson <benson31@llnl.gov>
This includes a fix from @tbennun.
DistData also includes the datatype, and so spuriously indicated things were sharded when mixing datatypes.
Co-authored-by: Tom Benson <benson31@llnl.gov>
Co-authored-by: Tom Benson <benson31@llnl.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an initial pass at AMP. It appears correct in the cases I've tested.
It has some particularly pointy bits in batchnorm and in gradient unscaling.