support excluded_layers for amp.decorate #52871
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
zhangting2020
force-pushed
the
amp_decorate
branch
from
db9d53a
to
e93bd5d
Compare
Xreki
reviewed
Apr 13, 2023
python/paddle/amp/auto_cast.py
Outdated
| ), | ||
| ): | ||
| need_keep_fp32 = True | ||
| elif (layer._dtype == 'float16') or isinstance( |
There was a problem hiding this comment.
layer._dtype代表什么，需要加一些注释解释下？这个接口也用于BF16吗，那BF16的LayerNorm参数是FP32还是BF16？
There was a problem hiding this comment.
已增加注释。该接口就是要统一fp16和bf16的参数转换过程。
bf16下原始接口为pure_bf16_initialize，会将所有层参数做转换，PR中处理为只有bn保持fp32，其他层都会被转换。
python/paddle/amp/auto_cast.py
Outdated
| # initialize parameters of the model | ||
| for idx in range(len(excluded_layers)): | ||
| for layer in excluded_layers[idx].sublayers(include_self=True): | ||
| layer._cast_to_low_precison_amp = False |
python/paddle/amp/auto_cast.py
Outdated
| ), | ||
| ): | ||
| need_keep_fp32 = True | ||
| elif not layer._cast_to_low_precison_amp: |
There was a problem hiding this comment.
确认下，LayerNorm是否可以通过某种方式，配置参数使用FP16/BF16类型？实际业务中可能会有这种需求，且pytorch的LayerNorm参数类型默认也是FP16的。
There was a problem hiding this comment.
目前实现不改变fp16的默认行为，依然是原来的处理方式，即BN、LayerNorm、InstanceNorm都保持fp32。
暂时没有提供任何方式设置fp16下允许LN的参数使用fp16。
python/paddle/amp/auto_cast.py
Outdated
| gradients will be FP32 dtype after the backpropagation. Default is False. | ||
| master_grad(bool, optional): For level='O2', whether to use float32 weight gradients for calculations such as gradient clipping, weight decay, and weight updates. If master_grad is enabled, the weight | ||
| gradients will be float32 dtype after the backpropagation. Default is False. | ||
| excluded_layers(Layer|list of Layer, optional): Specifies the layers not to be decorated. The weights of these layers will always keep float32 when level is O2. Default is None, the weights of the whole model will be casted to float16 or bfloat16. |
There was a problem hiding this comment.
确认下，这里是设置类型如[nn.LayerNorm]，还是设置实例对象如[norm]，还是两者皆可？
test/amp/test_amp_decorate.py
Outdated
| models=model, | ||
| level='O2', | ||
| dtype='float16', | ||
| excluded_layers=[model.conv], |
zhangting2020
force-pushed
the
amp_decorate
branch
3 times, most recently
from
257c7b6
to
49d9b84
Compare
zhangting2020
force-pushed
the
amp_decorate
branch
from
49d9b84
to
20c616b
Compare
Xreki
previously approved these changes
Apr 18, 2023
| layer, | ||
| ( | ||
| paddle.incubate.nn.FusedFeedForward, | ||
| paddle.incubate.nn.FusedMultiHeadAttention, | ||
| ), | ||
| ): | ||
| layer._amp_decorate(dtype='float16') | ||
| layer._amp_decorate(dtype=dtype) |
python/paddle/amp/auto_cast.py
Outdated
| master_grad(bool, optional): For level='O2', whether to use float32 weight gradients for calculations such as gradient clipping, weight decay, and weight updates. If master_grad is enabled, the weight | ||
| gradients will be float32 dtype after the backpropagation. Default is False. | ||
| excluded_layers(Layer|list of Layer, optional): Specifies the layers not to be decorated. The weights of these layers will always keep float32 when level is O2. `excluded_layers` can be specified as | ||
| an Layer instance/type or a list of Layer instances/tpyes. Default is None, the weights of the whole model will be casted to float16 or bfloat16. |
Xreki
approved these changes
Apr 18, 2023
jjyaoao
pushed a commit
to jjyaoao/Paddle
that referenced
this pull request
Apr 19, 2023
lijialin03
pushed a commit
to lijialin03/Paddle
that referenced
this pull request
Apr 25, 2023
zhangting2020
added a commit
to zhangting2020/Paddle
that referenced
this pull request
Apr 28, 2023
Xreki
pushed a commit
to Xreki/Paddle
that referenced
this pull request
May 2, 2023
zhangting2020
added a commit
to zhangting2020/Paddle
that referenced
this pull request
May 22, 2023
gongweibao
pushed a commit
that referenced
this pull request
May 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New featuresPR changes
APIsDescribe
support excluded_layers for amp.decorate英文文档：http://preview-paddle-pr-52871.paddle-docs-preview.paddlepaddle.org.cn/documentation/docs/en/api/paddle/amp/decorate_en.html
中文文档：PaddlePaddle/docs#5792