NLPDDPPlugin find_unused_parameters is configurable #3478
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Michelle Gill <mgill@nvidia.com>
ericharper
reviewed
Feb 3, 2022
fayejf
pushed a commit
that referenced
this pull request
Mar 2, 2022
* NLPDDPPlugin find_unused_parameters is configurable Signed-off-by: Michelle Gill <mgill@nvidia.com> * Choose simple example * Update nlp_overrides.py Co-authored-by: Eric Harper <complex451@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes
find_unused_parametersuser configurable inNLPDDPPluginby setting a default dictionary with the current value (False) and updating that dictionary with user provided kwargs. This ensures that the value is alwaysFalse, even if it changes from its current default (alsoFalse) in PyTorch'sDistributedDataParallel.An alternative option would be to remove it from the initialization call to
DistributedDataParallelsince it is alreadyFalseby default in PyTorch.The reason for this request is that
find_unused_parametersmust beTruefor MegaMolBART.I have read the contribution guidelines. All tests that are currently passing in
mainpass with this commit. (Note that two tests fail in NeMomain.) The code has been formatted with black and the commit is signed. This change was discussed with @ericharper several months ago.Signed-off-by: Michelle Gill mgill@nvidia.com