【Hackathon No61】add bf16 for mode #53195

Difers · 2023-04-21T14:03:56Z

PR types

Others

PR changes

APIs

Description

为mode算子添加fp16,bf16支持并添加单测
改动说明：

原先单测中out计算函数_mode1D不正确,导致众数计算结果都会说序列第一个,但由于原先input中重复值概率很小，因此未检测到。
对于众数bf16类型的计算结果测试，需要对输入对齐到精度损失后的input，例如input为[1.231,1.242,1.243],dtype=float32，计算众数，结果可以为1.231,index=0，而对bf16测试时,对input进行fp32 to uint16转换后,再转回，input精度损失后假设为[1.23,1.24,1.24]，众数计算结果应为1.24,index=1，不应与原本fp32的结果进行对照检测，因此对input 进行了fp32 to uint16, uint16 to fp32的精度损失转换。
众数的反向计算采用默认数值微分的方式会有误，故采用了user_defined_grads

paddle-bot · 2023-04-21T14:04:02Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-bot · 2023-04-21T14:04:06Z

❌ The PR is not created using PR's template. You can refer to this Demo.
Please use PR's template, it helps save our maintainers' time so that more developers get helped.

ZzSean · 2023-04-25T03:31:18Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+        self.outputs = {'Out': output, 'Indices': indices}
+        self.init_numeric_grads()
+
+    def init_numeric_grads(self):


这个函数是否可以提取出来，减少重复代码

ZzSean · 2023-04-25T03:32:54Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+        self.python_api = paddle.mode
+        self.dtype = np.float16
+        self.input_data = (
+            np.random.rand(2, 64, 1).astype(np.float16).astype(np.float32)


fp16应该不用这样转，初始化为fp16时，内部参考结果计算时会转回fp32的

ZzSean · 2023-04-28T02:35:19Z

python/paddle/fluid/tests/unittests/test_mode_op.py

@@ -56,6 +61,24 @@ def cal_mode(a, axis, keepdim=False):
    return modes, indexes


+def init_numeric_grads(input_shape, out_indices, axis):
+    grad = np.zeros(input_shape).astype(np.float32)


TestModeOp是float64的测试，但是这里cast为float32，属于高精度转低精度，不太合适，所以fp64这个单测是不是不用修改为user_defind_grads？

ZzSean · 2023-04-28T02:36:50Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+        self.init_last_dim_numeric_grads()
+
+    def init_last_dim_numeric_grads(self):
+        self.grad = np.zeros(self.input_data.shape).astype(np.float32)


这个也是同样的问题，fp64的测例无需修改为user_defined_grads，然后补充fp16和bf16的测例，使用user_defined_grads

这里反向不是精度的问题而是数值微分计算方式会有有问题，比如，[1,2,2]，众数是2，反向结果应该是[0,1,1]，而数值微分计算是:
(ypos,yneg是对应index的x加或减了delta后计算众数的结果)
index0: ypos=2,yneg=2,grad0=0;
index1:ypos=f([1,2+0.005,2])=1,yneg同理=1,grad1=0;
index2同理可得grad2=0,
导致反向结果为[0,0,0]，不符

了解，但是仍有两个问题：

对于double类型使用float32表示是否不合适

仍未补充fp16和bf16的TestModeOpLastdim测例

paddle-ci-bot · 2023-05-04T03:22:39Z

Sorry to inform you that ab60ca0's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

Difers · 2023-05-06T07:15:15Z

@ZzSean done~有空辛苦再review下

ZzSean · 2023-05-10T07:28:54Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+@unittest.skipIf(
+    not core.is_compiled_with_cuda(), "core is not compiled with CUDA"
+)
+class TestModeFP16Op(OpTest):


fp16的单测与原单测只有数据类型不同，和init_numeric_grads传的数据类型不同，能否把公共部分提出来，精简下代码，当前有很多重复代码

ZzSean · 2023-05-10T07:29:42Z

python/paddle/fluid/tests/unittests/test_mode_op.py

        self.attrs = {'axis': self.axis}
        output, indices = cal_mode(self.input_data, axis=self.axis)
        self.outputs = {'Out': output, 'Indices': indices}

    def test_check_output(self):
        paddle.enable_static()
-        self.check_output()
+        place = core.CUDAPlace(0)


这里可以直接使用check_output，可以直接继承TestModeOp，复用其中的函数

这里还没改，可以直接继承，不用再写test_check_output和test_check_grad

ZzSean · 2023-05-10T07:30:06Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+            self.input_data.shape,
+            self.outputs['Indices'],
+            self.axis,
+            np.float32,


减少这个传参，改为在内部判断

ZzSean · 2023-05-10T07:32:07Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+    def init_args(self):
+        self.axis = -1
+
+    def setUp(self):


同样的，lastdim与原case也只有shape和axis不同，也可以将shape的设置写在init_args中，或者新写一个函数，不用重复再写setUp中的代码

ZzSean · 2023-05-11T08:52:05Z

python/paddle/fluid/tests/unittests/test_mode_op.py

+
+class TestModeOpLastdim(TestModeOp):
+    def init_args(self):
+        self.axis = -1


我之前的意思是说把shape的设置也提出来，但是这里继承了TestModeOp后，测试的shape仍是(2, 64, 1)，并不是(2, 1, 1, 2, 30)了

Difers · 2023-05-12T12:50:47Z

@ZzSean done~,辛苦有空再review下

ZzSean

LGTM

paddle-bot bot added contributor External developers status: proposed labels Apr 21, 2023

luotao1 assigned luotao1, Vvsmile, ZzSean and Ligoml Apr 22, 2023

Difers force-pushed the add_fp_bf_seg_mode branch 5 times, most recently from 1a7352f to 14382b7 Compare April 24, 2023 02:24

ZzSean reviewed Apr 25, 2023

View reviewed changes

Difers force-pushed the add_fp_bf_seg_mode branch from 14382b7 to ab60ca0 Compare April 26, 2023 06:12

ZzSean reviewed Apr 28, 2023

View reviewed changes

Difers force-pushed the add_fp_bf_seg_mode branch 2 times, most recently from e937e26 to d9ae910 Compare May 5, 2023 14:55

ZzSean reviewed May 10, 2023

View reviewed changes

Difers force-pushed the add_fp_bf_seg_mode branch from d9ae910 to 2079c32 Compare May 11, 2023 05:23

ZzSean reviewed May 11, 2023

View reviewed changes

Difers force-pushed the add_fp_bf_seg_mode branch 2 times, most recently from ab409ce to 63e709f Compare May 12, 2023 06:10

Difers added 12 commits May 12, 2023 07:18

add bf16 for mode

6e501ff

remove random seed 666

84da7d1

try to fix op_type error

94176c8

test for me

e003ac1

try to fix op_type

b94f005

fix redundancy code

372741a

add fp,bf for lastdim

98d7eb4

fix some error

4d45605

simplify code

112ba4e

fix shape error

0239f61

optype error

1318e92

fix skipif bf16

39b8756

Difers force-pushed the add_fp_bf_seg_mode branch from 63e709f to 39b8756 Compare May 12, 2023 07:18

ZzSean approved these changes May 16, 2023

View reviewed changes

ZzSean merged commit 640cff0 into PaddlePaddle:develop May 16, 2023

luotao1 changed the title ~~【Hackathon No57】add bf16 for mode~~ 【Hackathon No61】add bf16 for mode May 16, 2023

Difers mentioned this pull request May 18, 2023

【PaddlePaddle Hackathon 第四期】任务总览 #51281

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【Hackathon No61】add bf16 for mode #53195

【Hackathon No61】add bf16 for mode #53195

Difers commented Apr 21, 2023 •

edited

Loading

paddle-bot bot commented Apr 21, 2023

paddle-bot bot commented Apr 21, 2023

ZzSean Apr 25, 2023

ZzSean Apr 25, 2023

ZzSean Apr 28, 2023

ZzSean Apr 28, 2023

Difers Apr 28, 2023 •

edited

Loading

ZzSean May 5, 2023

paddle-ci-bot bot commented May 4, 2023

Difers commented May 6, 2023

ZzSean May 10, 2023

ZzSean May 10, 2023

ZzSean May 11, 2023

ZzSean May 10, 2023

ZzSean May 10, 2023

ZzSean May 11, 2023

Difers commented May 12, 2023

ZzSean left a comment

【Hackathon No61】add bf16 for mode #53195

【Hackathon No61】add bf16 for mode #53195

Conversation

Difers commented Apr 21, 2023 • edited Loading

PR types

PR changes

Description

paddle-bot bot commented Apr 21, 2023

paddle-bot bot commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Difers Apr 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paddle-ci-bot bot commented May 4, 2023

Difers commented May 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Difers commented May 12, 2023

ZzSean left a comment

Choose a reason for hiding this comment

Difers commented Apr 21, 2023 •

edited

Loading

Difers Apr 28, 2023 •

edited

Loading