Refine param conversion logic in layer.to #36862
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhangbo9674
changed the title
zhiqiu
reviewed
Nov 8, 2021
| else: | ||
| size_dtype = core.size_of_dtype(dtype) | ||
| waiting_alloc_memory = ( | ||
| (t.numel().numpy()[0] * size_dtype) / 256 + 1) * 256 * 1.2 |
| @@ -121,8 +121,8 @@ def __init__(self, name_scope=None, dtype="float32"): | |||
| self._forward_pre_hooks = collections.OrderedDict() | |||
| self._forward_post_hooks = collections.OrderedDict() | |||
|
|
|||
| self._parameters_transform_map = {} | |||
| self._buffers_transform_map = {} | |||
| #self._parameters_transform_map = {} | |||
zhangbo9674
added a commit
to zhangbo9674/Paddle
that referenced
this pull request
Dec 12, 2021
* refine layer to * delete comment * refine logic * refine code * refine pure_fp16_init * refine comment
zhangbo9674
added a commit
to zhangbo9674/Paddle
that referenced
this pull request
Dec 12, 2021
* refine layer to * delete comment * refine logic * refine code * refine pure_fp16_init * refine comment
zhangbo9674
added a commit
to zhangbo9674/Paddle
that referenced
this pull request
Dec 13, 2021
* refine layer to * delete comment * refine logic * refine code * refine pure_fp16_init * refine comment
lanxianghit
pushed a commit
that referenced
this pull request
Dec 13, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
APIs
Describe
Class Layer的layer.to()用于将网络参数做device或dtype的转换。1、原始数据处理逻辑：
（1）遍历Layer的参数、梯度、buffer；
（2）拷贝到指定device：
new_t = t._copy_to(device, blocking)；（3）数据类型转换：
new_t = new_t.cast(dtype=dtype)；（4）返回new_t；
2、存在的问题及需求：
在fp16训练模式下，通过layer.to方法将网络参数从fp32转为fp16，利用上述逻辑在第(3)步的阶段，显存会同时持有fp32和fp16两份参数，导致显存增长。
3、修改后的逻辑：
以float32网络通过layer.to(‘float16’)将参数转为float16为例：
（1）判断显存是否足够再创建一个参数；
（2）如果显存充足，则直接将数据cast为float16。
（3）如果显存不够，则将参数拷贝到cpu并释放gpu持有的tensor，在cpu将数据cast为float16，最后将转换后的数据转回gpu；
程序运行流程图如下图所示：