【Hackathon No57】add_fp16_bf16_for_dot & bf16_for_cross #52426
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
Difers
force-pushed
the
add_fp_bf_for_dot_cross
branch
2 times, most recently
from
f37a82c to
ec50629
Compare
| self.python_api = paddle.cross | ||
| self.initTestCase() | ||
| self.inputs = { | ||
| 'X': np.random.random(self.shape).astype(self.dtype), |
There was a problem hiding this comment.
BF16类型的输入和输出需要调用convert_float_to_uint16函数做类型转换
| for i in range(1024): | ||
| z_list.append(np.cross(x[i], y[i])) | ||
| self.outputs = { | ||
| 'Out': np.array(z_list).astype(np.float32).reshape(self.shape) |
There was a problem hiding this comment.
BF16类型的输入和输出需要调用convert_float_to_uint16函数做类型转换
| 'X': convert_float_to_uint16(self.x), | ||
| 'Y': convert_float_to_uint16(self.y), | ||
| } | ||
| self.outputs = {'Out': self.out} |
There was a problem hiding this comment.
输出也需要调用convert_float_to_uint16进行类型转换
| if core.is_compiled_with_cuda(): | ||
| place = core.CUDAPlace(0) | ||
| if core.is_bfloat16_supported(place): | ||
| self.check_grad_with_place(place, ['X', 'Y'], 'Out') |
There was a problem hiding this comment.
按照数学期望来计算，这里的前向atol可能需要设置为0.5
Difers
force-pushed
the
add_fp_bf_for_dot_cross
branch
from
ec50629 to
72d731f
Compare
|
LGTM |
| ) | ||
|
|
||
| def init_input_output(self): | ||
| self.x = np.random.uniform(0.1, 1, [121]).astype(self.dtype) |
There was a problem hiding this comment.
这里改动的时候发现，dot的bf16计算反向时，过程中经过fp32转uint16，又经过fp16转32,观察其数值在100+的时候小数点后面精度损失有的比较严重（具体是在计算get_numeric_gradient，y_pos,y_neg因精度损失相同了，导致数值微分是0），因此后续改动选择采用了user_defined_grads
| self.initTestCase() | ||
| self.inputs = { | ||
| 'X': convert_float_to_uint16( | ||
| np.random.random(self.shape).astype(self.dtype) |
Difers
force-pushed
the
add_fp_bf_for_dot_cross
branch
from
72d731f to
fd01944
Compare
| place, | ||
| ['X', 'Y'], | ||
| 'Out', | ||
| user_defined_grads=[self.y / 11.0, self.x / 11.0], |
There was a problem hiding this comment.
可以用user_defined_grads，但是不要用11.0这种magic number，用shape这种参数来代替
Difers
force-pushed
the
add_fp_bf_for_dot_cross
branch
from
fd01944 to
8bb286d
Compare
|
@ZzSean done，辛苦再review下~ |
PR types
Others
PR changes
APIs
Describe
为dot 注册FP16,BF16， 完善单测
为cross 注册BF16， 完善单测