Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【PIR API adaptor No.161、162】Migrate paddle.vision.ops.nms paddle.nn.functional.one_hot into pir #58735

Merged
merged 10 commits into from
Dec 6, 2023

Conversation

ooooo-create
Copy link
Contributor

@ooooo-create ooooo-create commented Nov 6, 2023

PR types

Others

PR changes

APIs

Description

No.161-162

  • 新IR Python API适配升级 #58067

  • paddle.vision.ops.nms 单测覆盖率 2/2

  • paddle.nn.functional.one_hot 单测覆盖率 3/13
    BadInputTestOnehotV2” 当前不支持
    目前pir的OpTest逻辑还不是支持lod的处理。所以one_hot继承自OpTest的单测先不用适配了

@paddle-bot paddle-bot bot added the contributor External developers label Nov 6, 2023
@luotao1 luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Nov 7, 2023
@MarioLulab
Copy link
Contributor

test_nn_functional_hot_op.py 里出现的报错:

AssertionError: Detect there is KernelSignature for `one_hot_v2` op, please set the `self.python_api` if you set check_dygraph = True

是因为没有在继承自 OpTest 的单测中添加 python_api 属性,需要在 setUp 中配置一下 self.python_api

@ooooo-create
Copy link
Contributor Author

test_nn_functional_hot_op.py 里出现的报错:

AssertionError: Detect there is KernelSignature for `one_hot_v2` op, please set the `self.python_api` if you set check_dygraph = True

是因为没有在继承自 OpTest 的单测中添加 python_api 属性,需要在 setUp 中配置一下 self.python_api

添加之后出现和 test_one_hot_v2_op 一样的报错

1265:  .E...EEE
1265: ======================================================================
1265: ERROR: test_check_output (test_nn_functional_hot_op.TestOneHotOp)
1265: ----------------------------------------------------------------------
1265: Traceback (most recent call last):
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 49, in test_check_output
1265:     self.check_output(check_dygraph=False, check_pir=True)
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 2686, in check_output
1265:     check_pir=check_pir,
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 2548, in check_output_with_place
1265:     pir_checker.check()
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 2149, in check
1265:     self.calculate_output()
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 2335, in calculate_output
1265:     pir_outs = self.op_test._calc_pir_output(place)
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 1330, in _calc_pir_output
1265:     ) = self.get_ir_input_attr_dict_and_feed(stop_gradient=True)
1265:   File "/paddle/build/test/legacy_test/op_test.py", line 1267, in get_ir_input_attr_dict_and_feed
1265:     if OpTestUtils.is_bfloat16_type(tup[1].dtype)
1265: IndexError: list index out of range

@0x45f
Copy link
Contributor

0x45f commented Nov 9, 2023

对于one_hot的单测:单测文件内继承自OpTest的单测因为配置了x_lod参数,目前pir的OpTest逻辑还不是支持lod的处理。所以one_hot继承自OpTest的单测先不用适配了,先把继承自unittest.TestCase的单测适配一下~

@ooooo-create
Copy link
Contributor Author

对于one_hot的单测:单测文件内继承自OpTest的单测因为配置了x_lod参数,目前pir的OpTest逻辑还不是支持lod的处理。所以one_hot继承自OpTest的单测先不用适配了,先把继承自unittest.TestCase的单测适配一下~

thanks~适配继承unittest.TestCase的单测有报错

1265: ======================================================================
1265: ERROR: test_api_with_depthTensor (test_nn_functional_hot_op.TestOneHotOpApi)
1265: ----------------------------------------------------------------------
1265: Traceback (most recent call last):
1265:   File "/paddle/build/python/paddle/pir_utils.py", line 115, in impl
1265:     func(*args, **kwargs)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 136, in test_api_with_depthTensor
1265:     self._run(num_classes)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 165, in _run
1265:     return_numpy=False,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1644, in run
1265:     return_numpy=return_numpy,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1947, in _run_pir_impl
1265:     scope,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1037, in get_pir_program_and_executor
1265:     program, fetch_list=fetch_list, fetch_var_name=fetch_var_name
1265:   File "/paddle/build/python/paddle/base/executor.py", line 511, in _add_pir_fetch_ops
1265:     global_block, fetch_list, fetch_var_name, fetch_op
1265:   File "/paddle/build/python/paddle/base/executor.py", line 430, in has_fetch_operations
1265:     "There is a fetch op in Program which will fetch variable that is not belong to fetch_targets."
1265: Exception: There is a fetch op in Program which will fetch variable that is not belong to fetch_targets.
1265: 
1265: ======================================================================
1265: FAIL: test_error (test_nn_functional_hot_op.BadInputTestOnehotV2)
1265: ----------------------------------------------------------------------
1265: Traceback (most recent call last):
1265:   File "/paddle/build/python/paddle/pir_utils.py", line 115, in impl
1265:     func(*args, **kwargs)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 183, in test_error
1265:     self.assertRaises(TypeError, test_bad_x)
1265: AssertionError: TypeError not raised by test_bad_x
1265: 

@MarioLulab
Copy link
Contributor

对于one_hot的单测:单测文件内继承自OpTest的单测因为配置了x_lod参数,目前pir的OpTest逻辑还不是支持lod的处理。所以one_hot继承自OpTest的单测先不用适配了,先把继承自unittest.TestCase的单测适配一下~

thanks~适配继承unittest.TestCase的单测有报错

1265: ======================================================================
1265: ERROR: test_api_with_depthTensor (test_nn_functional_hot_op.TestOneHotOpApi)
1265: ----------------------------------------------------------------------
1265: Traceback (most recent call last):
1265:   File "/paddle/build/python/paddle/pir_utils.py", line 115, in impl
1265:     func(*args, **kwargs)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 136, in test_api_with_depthTensor
1265:     self._run(num_classes)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 165, in _run
1265:     return_numpy=False,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1644, in run
1265:     return_numpy=return_numpy,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1947, in _run_pir_impl
1265:     scope,
1265:   File "/paddle/build/python/paddle/base/executor.py", line 1037, in get_pir_program_and_executor
1265:     program, fetch_list=fetch_list, fetch_var_name=fetch_var_name
1265:   File "/paddle/build/python/paddle/base/executor.py", line 511, in _add_pir_fetch_ops
1265:     global_block, fetch_list, fetch_var_name, fetch_op
1265:   File "/paddle/build/python/paddle/base/executor.py", line 430, in has_fetch_operations
1265:     "There is a fetch op in Program which will fetch variable that is not belong to fetch_targets."
1265: Exception: There is a fetch op in Program which will fetch variable that is not belong to fetch_targets.
1265: 
1265: ======================================================================
1265: FAIL: test_error (test_nn_functional_hot_op.BadInputTestOnehotV2)
1265: ----------------------------------------------------------------------
1265: Traceback (most recent call last):
1265:   File "/paddle/build/python/paddle/pir_utils.py", line 115, in impl
1265:     func(*args, **kwargs)
1265:   File "/paddle/build/test/legacy_test/test_nn_functional_hot_op.py", line 183, in test_error
1265:     self.assertRaises(TypeError, test_bad_x)
1265: AssertionError: TypeError not raised by test_bad_x
1265: 
  1. test_api_with_depthTensor:问题的原因应该和 PIR Python API 适配升级任务的 bug 修复手册 #58259 (comment) 是一样的,按照这个修改吧
  2. FAIL: test_error (test_nn_functional_hot_op.BadInputTestOnehotV2): 先跳过 test_errors 的 pir 测试吧,并且在 PR 描述里说明一下

Copy link

paddle-ci-bot bot commented Nov 19, 2023

Sorry to inform you that 7dbb662's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

@MarioLulab
Copy link
Contributor

MarioLulab commented Nov 23, 2023

当前问题可以按上述方式修复 😄

@ooooo-create
Copy link
Contributor Author

当前问题可以按上述方式修复 😄

好哒好哒~

Copy link
Contributor

@MarioLulab MarioLulab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test/legacy_test/test_zero_dim_tensor.py 文件里还有 TestNoBackwardAPIStatic.test_one_hot_label 单测没有迁移,麻烦迁移一下吧~

test/legacy_test/test_ops_nms.py Outdated Show resolved Hide resolved
test/legacy_test/test_one_hot_v2_op.py Outdated Show resolved Hide resolved
python/paddle/vision/ops.py Outdated Show resolved Hide resolved
python/paddle/vision/ops.py Show resolved Hide resolved
test/legacy_test/test_ops_nms.py Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Show resolved Hide resolved
test/legacy_test/test_one_hot_v2_op.py Show resolved Hide resolved
@ooooo-create
Copy link
Contributor Author

感谢这么详细的指导~有个问题,test_one_hot_v2_op 里的 label.desc.set_need_check_feed(False)我注释掉,然后aistudio上验证没有报错,但是 ci 里好像说
image
是不是因为我注释掉那句话了,还是为什么,然后不注释掉,本地aistudio 上会有下面这个问题
image

@0x45f
Copy link
Contributor

0x45f commented Dec 1, 2023

感谢这么详细的指导~有个问题,test_one_hot_v2_op 里的 label.desc.set_need_check_feed(False)我注释掉,然后aistudio上验证没有报错,但是 ci 里好像说 image 是不是因为我注释掉那句话了,还是为什么,然后不注释掉,本地aistudio 上会有下面这个问题 image

b69dd80cdf6498175c3ac48ef3ba09d9
这个问题改一下单测的代码吧,在老的静态图下才进行set_need_check_feed,在pir下把label_data 转换成int64类型~

@ooooo-create
Copy link
Contributor Author

感谢这么详细的指导~有个问题,test_one_hot_v2_op 里的 label.desc.set_need_check_feed(False)我注释掉,然后aistudio上验证没有报错,但是 ci 里好像说 image 是不是因为我注释掉那句话了,还是为什么,然后不注释掉,本地aistudio 上会有下面这个问题 image

b69dd80cdf6498175c3ac48ef3ba09d9 这个问题改一下单测的代码吧,在老的静态图下才进行set_need_check_feed,在pir下把label_data 转换成int64类型~

好哒~,done

Copy link
Contributor

@MarioLulab MarioLulab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

麻烦更新一下 paddle.nn.functional.one_hot 的单测覆盖率。并且在 pr 描述中将已经解决问题的报错栈删除,方便我们后续排查~

test/legacy_test/test_ops_nms.py Outdated Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Outdated Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Outdated Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Outdated Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Outdated Show resolved Hide resolved
test/legacy_test/test_nn_functional_hot_op.py Outdated Show resolved Hide resolved
Copy link
Contributor

@MarioLulab MarioLulab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

python/paddle/vision/ops.py Outdated Show resolved Hide resolved
test/legacy_test/test_ops_nms.py Outdated Show resolved Hide resolved
@0x45f 0x45f merged commit e8eb917 into PaddlePaddle:develop Dec 6, 2023
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contributor External developers HappyOpenSource 快乐开源活动issue与PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants