[Paddle-ASP]Support sharding training for the Nvidia's ASP(2:4 sparsity) functionality #37725
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sync repo
|
✅ This PR's description meets the template requirements! |
|
Thanks for your contribution! |
qingqing01
requested review from
wanghaoshuang,
qingqing01,
JZ-LIANG and
Xreki
| @@ -150,7 +155,8 @@ def prune_model(main_program=None, | |||
| n=2, | |||
| m=4, | |||
| mask_algo='mask_1d', | |||
| with_mask=True): | |||
| with_mask=True, | |||
| sharding=False): | |||
There was a problem hiding this comment.
可以让用户直接传一个place么？那种方式更好理解？
There was a problem hiding this comment.
理解是好理解，但是我们需要给用户额外说明。
此外，即使说明了，也有经常place设置错误的风险，然后出bug。
我的理解是，place在prune_model里面，代码稳定性更高一些呢？
|
Sorry to inform you that 13083b1's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually. |
added tests for optimizer compatibility and modified prune_model API. |
|
请修改下PR标题，方便后续检索和管理自己的工作。 |
Done, thanks |
minghaoBD
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Bug fixes
PR changes
Others
Describe
Nvidia has implemented 2:4 sparsity code in PaddlePaddle, supporting fleet distributed training. But when we are trying to train with sharding strategy (the model parallel paradigm in PaddlePaddle), GPU:0 will always be OOM while other GPUs seems normal.
After fix, developers should pass in an argument: shading=True when calling sparsity.prune_model() with the sharding strategy. Otherwise no difference when using the APIs.