[Bug]RuntimeError: CumSum layer with name 'p2o.CumSum.0'

[JiehangXie]

System information (version)

fastspeech2 model trained by PaddlePaddle convert to OpenVINO
I have tried to use both paddle and onnx format model.
When used paddle format model, it will cause:
vector too long.

When used ONNX format model, it will cause:
"CumSum layer with name 'p2o.CumSum.0' doesn't support 'axis' input tensor with non scalar rank"

All models and inference code has been uploaded to BaiduNetdisk and Google Drive:
https://pan.baidu.com/s/1u_aNmiWz8UflW3l2m5cSCg?pwd=quck
https://drive.google.com/file/d/1ci6wMuPb6IWLhkBK7CpiNH8KSUAb-35Y/view?usp=sharing
Thank you you all to have a check!

2023.3.8: try #14961 and #14993, it seems not work.
RuntimeError: CPU plug-in doesn't support Squeeze node with inputs num equal 1

[meiyang-intel]

1. Have tried 'fastspeech2_canton.pdmodel'. And check the pdmodel.
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ ERROR ] Check 'creator_it != CREATORS_MAP.end()' failed at src/frontends/paddle/src/frontend.cpp:46: FrontEnd API failed with OpConversionFailure: : No creator found for sin node.
   Currently some op in this pdmodel such as 'sin', 'cos', 'set_value' are not supported in the OpenVINO Paddle frontend.

2. also tried the 'fastspeech2_canton.xml'. Encounter the same issue of 'cumsum'
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ INFO ] Read model took 356.38 ms [ INFO ] Original model I/O parameters: [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [...] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [...] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 5/11] Resizing model to match image sizes and given batch [ INFO ] Number of test configurations is calculated basing on -data_shape parameter [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ INFO ] Reshaping model: 'cast_0.tmp_0': [?], 'spk_id': [1] [ INFO ] Reshape model took 118.90 ms [Step 6/11] Configuring input of the model [ INFO ] Model batch size: 1 [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [N] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [C] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 7/11] Loading the model to the device [ ERROR ] CumSum layer with name 'p2o.CumSum.0' doesn't support 'axis' input tensor with non scalar rank
   Check the 'cumsum' in onnx and pdmodel that the axis value is '-1' and OpenVINO recognize it as dynamic shape. The dynamic shape of 'axis' in 'cumsum' is not supported in OpenVINO. So that's why it report this error.

[JiehangXie]

1. Have tried 'fastspeech2_canton.pdmodel'. And check the pdmodel.
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ ERROR ] Check 'creator_it != CREATORS_MAP.end()' failed at src/frontends/paddle/src/frontend.cpp:46: FrontEnd API failed with OpConversionFailure: : No creator found for sin node.
   Currently some op in this pdmodel such as 'sin', 'cos', 'set_value' are not supported in the OpenVINO Paddle frontend.
2. also tried the 'fastspeech2_canton.xml'. Encounter the same issue of 'cumsum'
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ INFO ] Read model took 356.38 ms [ INFO ] Original model I/O parameters: [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [...] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [...] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 5/11] Resizing model to match image sizes and given batch [ INFO ] Number of test configurations is calculated basing on -data_shape parameter [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ INFO ] Reshaping model: 'cast_0.tmp_0': [?], 'spk_id': [1] [ INFO ] Reshape model took 118.90 ms [Step 6/11] Configuring input of the model [ INFO ] Model batch size: 1 [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [N] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [C] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 7/11] Loading the model to the device [ ERROR ] CumSum layer with name 'p2o.CumSum.0' doesn't support 'axis' input tensor with non scalar rank
   Check the 'cumsum' in onnx and pdmodel that the axis value is '-1' and OpenVINO recognize it as dynamic shape. The dynamic shape of 'axis' in 'cumsum' is not supported in OpenVINO. So that's why it report this error.

Do you have any solution if I insist keeping the model unchanged?

[meiyang-intel]

1. Have tried 'fastspeech2_canton.pdmodel'. And check the pdmodel.
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ ERROR ] Check 'creator_it != CREATORS_MAP.end()' failed at src/frontends/paddle/src/frontend.cpp:46: FrontEnd API failed with OpConversionFailure: : No creator found for sin node.
   Currently some op in this pdmodel such as 'sin', 'cos', 'set_value' are not supported in the OpenVINO Paddle frontend.
2. also tried the 'fastspeech2_canton.xml'. Encounter the same issue of 'cumsum'
   [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [Step 2/11] Loading OpenVINO Runtime [ INFO ] OpenVINO: [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] Device info: [ INFO ] CPU [ INFO ] Build ................................. 2023.0.0-9946-c09b2ff8b10 [ INFO ] [ INFO ] [Step 3/11] Setting device configuration [ WARNING ] Device(CPU) performance hint is set to THROUGHPUT [ WARNING ] Batch size is set. Auto batching will be disabled [Step 4/11] Reading model files [ INFO ] Loading model files [ INFO ] Read model took 356.38 ms [ INFO ] Original model I/O parameters: [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [...] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [...] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 5/11] Resizing model to match image sizes and given batch [ INFO ] Number of test configurations is calculated basing on -data_shape parameter [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ WARNING ] Input 'spk_id' doesn't have batch dimension in layout. -b option will be ignored for this input. [ INFO ] Reshaping model: 'cast_0.tmp_0': [?], 'spk_id': [1] [ INFO ] Reshape model took 118.90 ms [Step 6/11] Configuring input of the model [ INFO ] Model batch size: 1 [ INFO ] Network inputs: [ INFO ] cast_0.tmp_0 (node: text) : i64 / [N] / [?] [ INFO ] spk_id (node: spk_id) : i64 / [C] / [1] [ INFO ] Network outputs: [ INFO ] elementwise_add_82 (node: elementwise_add_82) : f32 / [...] / [?,80] [Step 7/11] Loading the model to the device [ ERROR ] CumSum layer with name 'p2o.CumSum.0' doesn't support 'axis' input tensor with non scalar rank
   Check the 'cumsum' in onnx and pdmodel that the axis value is '-1' and OpenVINO recognize it as dynamic shape. The dynamic shape of 'axis' in 'cumsum' is not supported in OpenVINO. So that's why it report this error.

Do you have any solution if I insist keeping the model unchanged?

will think about how to fix it

[JiehangXie]

will think about how to fix it

@meiyang-intel Thanks！Hope OpenVINO can solve this problem quickly so that PaddleSpeech Models can be used with OpenVINO on Intel CPU.Keep in touch!

[yuxu42]

In order to support the PaddleSpeech model in OpenVINO, there are two way. One direct way is Paddle formatted model input (*pdmodel), and the other way is Paddle -> ONNX, and ONNX format as OV input. Currently, 3 ops (sine, cosine, setvalue) need to enable in Paddle Frontend in order to directly support Paddle format input.
ONNX may be the potential way to quickly solve the problem. @tomdol Could you please take a look? Thanks!

[yuxu42]

@JiehangXie Checked with PaddlePaddle team, Paddle operator of set_value (with step!=1) cannot be converted to ONNX op correctly with current paddle2onnx tool. It's a blocking issue of the inferring model in other framework/backend. Could you please also raise the ticket in PaddlePaddle community?

[JiehangXie]

@JiehangXie Checked with PaddlePaddle team, Paddle operator of set_value (with step!=1) cannot be converted to ONNX op correctly with current paddle2onnx tool. It's a blocking issue of the inferring model in other framework/backend. Could you please also raise the ticket in PaddlePaddle community?

@yuxu42 We have already converted .pdmodel and .pdiparams to ONNX model, and most of TTS models in PaddleSpeech can be converted successfully with Paddle2ONNX, you can try to use this model for test: https://drive.google.com/file/d/1ci6wMuPb6IWLhkBK7CpiNH8KSUAb-35Y/view?usp=sharing,

[xuewenyuan]

I met the same problem when compiling a DETR model. Have you solve it ?

net = ie.read_model(model='detr_r50_1x_coco.onnx')
compiled_model = ie.compile_model(net, 'CPU')

The onnx file is converted by paddle2onnx and the onnx can successfully run under onnxruntime

[xczhai]

@xuewenyuan

1. Could you help share the models including *.pdmodel and *.onnx?
2. Could you help share the commands of conversion using paddle2onnx?

That will help us address the issue quickly.

[xuewenyuan]

1. Packages

onnx==1.12.0
onnxruntime==1.14.1
openvino==2022.3.0
openvino-dev==2022.3.0
openvino-telemetry==2022.3.0
paddle2onnx==1.0.6
paddledet==2.6.0
paddlepaddle-gpu==2.4.2

2. Install PaddleDetection followed README

3. Export paddle model

Download official pretrained deformable detr (deformable_detr_r50_1x_coco.pdparams)
The path under "PaddleDetection"
The default output path is "PaddleDetection/output_inference/deformable_detr_r50_1x_coco"
python tools/export_model.py -c configs/deformable_detr/deformable_detr_r50_1x_coco.yml -o weights=deformable_detr_r50_1x_coco.pdparams

4. Convert to ONNX

paddle2onnx --model_dir output_inference/deformable_detr_r50_1x_coco --model_filename model.pdmodel --params_filename model.pdiparams --opset_version 11 --save_file deformable_detr_r50_1x_coco.onnx

5. Convert ONNX to OpenVINO IR

mo --input_model deformable_detr_r50_1x_coco.onnx

6. Run ONNX model in ONNX runtime (No problem)

from onnxruntime import InferenceSession
predictor = InferenceSession(onnx_file)
outputs = predictor.run(output_names=None, input_feed=inputs)

6. Run ONNX model in OpenVINO runtime （RuntimeError）

from openvino.runtime import Core
ie = Core()
model_onnx = ie.read_model(model=onnx_file)
compiled_model_onnx = ie.compile_model(model=model_onnx, device_name="CPU")

File ~/nas1/anaconda3/envs/openvino_env/lib/python3.8/site-packages/openvino/runtime/ie_api.py:399, in Core.compile_model(self, model, device_name, config)
--> 399     super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: CumSum layer with name 'p2o.CumSum.0'  doesn't support 'axis' input tensor with non scalar rank

7. Run ONNX model in OpenVINO runtime （RuntimeError）

from openvino.runtime import Core
ie = Core()
ir_path = 'deformable_detr_r50_1x_coco.xml'
model_ir = ie.read_model(model=ir_path)
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU

File ~/nas1/anaconda3/envs/openvino_env/lib/python3.8/site-packages/openvino/runtime/ie_api.py:399, in Core.compile_model(self, model, device_name, config)
--> 399     super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: CumSum layer with name 'p2o.CumSum.0'  doesn't support 'axis' input tensor with non scalar rank

@xczhai

[xczhai]

@xuewenyuan
A new PR is created to enable sin and cos for PaddlePaddle just now. #17525

Could you compile the openvino with the above PR and then try to load model.pdmodel?

how to compile: https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux.md

[xuewenyuan]

@xczhai
Thank you for the quick reply. It may be a problem to compile from the source because “sudo” is required. When will the sin and cos are enable in the release version.

[xczhai]

@xuewenyuan
#17525 can enable deformable_detr on local server. It still needs some time to review and merge. I will push this.
But the quickest way is recommended to build openvino with this PR.

[xuewenyuan]

@xczhai
I have tried to build your the "pd_enable_sin_and_cos" branch on my laptop (Windows10) followed the MD file. https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_windows.md
When I install the built wheel package "openvino_dev-2023.1.0-11001-py3-none-any.whl". A dependency error occurred:

ERROR: Could not find a version that satisfies the requirement openvino==2023.1.0 (from openvino-dev) (from versions: 2021.2, 2021.3.0, 2021.4.0, 2021.4.1, 2021.4.2, 2022.1.0.dev20220131, 2022.1.0.dev20220215, 2022.1.0.dev20220302, 2022.1.0.dev20220316, 2022.1.0, 2022.2.0.dev20220829, 2022.2.0, 2022.3.0.dev20221103, 2022.3.0.dev20221125, 2022.3.0, 2023.0.0.dev20230119, 2023.0.0.dev20230217, 2023.0.0.dev20230407, 2023.0.0.dev20230427)
ERROR: No matching distribution found for openvino==2023.1.0

[xczhai]

@xuewenyuan

• I tried it on ubuntu and didn't try it on windows. It is more reproduced on linux platform.
• If you build it on windows successfully, you can import it directly on windows. Just set some environment variables and don't need to install wheel. If the error still appears, it may be an environment problem on windows.

[xuewenyuan]

@xczhai
I have build your openvino branch on Windows, many warings but no errors. I have test previous deformable_detr model.

from openvino.tools import mo
from openvino.runtime import serialize, Core

# onnx_file = 'deformable_detr_r50_1x_coco/deformable_detr_r50_1x_coco.onnx'
paddle_file = 'deformable_detr_r50_1x_coco/model.pdmodel'

core = Core()
# directly load paddle file, successfully
ov_model_pd = core.read_model(paddle_file)
ov_modle_pd = core.compile_model(ov_model_pd, device_name="CPU")

# convert to ir and load, successfully
ov_model_ir = mo.convert_model(paddle_file)
serialize(ov_model_ir, str('deformable_detr_r50_1x_coco.xml'))
ov_model_ir = core.read_model('deformable_detr_r50_1x_coco.xml')
ov_modle_ir = core.compile_model(ov_model_ir, device_name="CPU")

However, an error occurred when I tested an image.

image_file = '000000014439.jpg' 
#https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.6/demo/000000014439.jpg
infer_cfg = 'deformable_detr_r50_1x_coco/infer_cfg.yml'

inputs = preprocess()
# "inputs" is a dict containing:
# 'im_shape': array([ 800., 1267.], dtype=float32), 
# 'scale_factor': array([1.980198, 1.980198], dtype=float32),
# 'image': array() with shape of (3, 800, 1267) 

results = ov_modle_ir([inputs])

Traceback (most recent call last):
  File "test.py", line 209, in <module>
    results = ov_model([inputs])
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\ie_api.py", line 266, in __call__
    return self._infer_request.infer(
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\ie_api.py", line 74, in infer
    return OVDict(super().infer(_data_dispatch(
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 349, in _data_dispatch
    return create_shared(inputs, request) if is_shared else create_copied(inputs, request)
  File "path_to_python\Python\Python38\lib\functools.py", line 875, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 177, in _
    return {k: value_to_tensor(v, request=request, is_shared=True, key=k) for k, v in request._inputs_data.items()}
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 177, in <dictcomp>
    return {k: value_to_tensor(v, request=request, is_shared=True, key=k) for k, v in request._inputs_data.items()}
  File "path_to_python\Python\Python38\lib\functools.py", line 875, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "path_to_openvino_install\openvino\python\python3.8\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 39, in value_to_tensor
    raise TypeError(f"Incompatible inputs of type: {type(value)}")
TypeError: Incompatible inputs of type: <class 'dict'>

onnxruntime has no problem for this inputs

[xczhai]

@xuewenyuan
Glad to know you build the OpenVINO and import PDPD model successfully.
From your description, some typo exists.

1. image shape should be 4-D instead of 3-D.
2. the execute code should be results = ov_modle_ir(inputs) instead of results = ov_modle_ir([inputs]).

[avitial]

Ref. 111483

[xuewenyuan]

@xczhai
Many thanks! Now, I have successfully run a pretrained PDPD model of Deformable DETR through a python interface.
The ONNX and IR files are verified. I will try a C++ interface and build my own model based on the PDPD Deformable DETR (e.g., HDETR). I hope that goes well.

[xczhai]

@xuewenyuan
That sounds great!

1. Also, cumsum problem is solved in enable 1-D axis in cumsum #17650.
   Both PDPD and ONNX will be available based these branches, I think. You can choose depending on your requirements.
2. Hope these docs can help you.
   https://docs.openvino.ai/nightly/openvino_deployment_guide.html
   https://docs.openvino.ai/nightly/notebooks/109-latency-tricks-with-output.html

If any problem, please reach us.

[xuewenyuan]

@xczhai
I have a new error when I read a custom paddle model that is based on DINO and Deformable DETR. How could I identify this problem and solve it?
When reading the DINO model, the same error occurs. (I did not test DINO before.)
should I open a new issue?

File "test.py", line 228, in <module>
    ov_model = core.read_model(paddle_file)
RuntimeError: Check 'false' failed at path_to_openvino\src\inference\src\core.cpp:100:
Check 'false' failed at path_to_openvino\src\frontends\paddle\src\frontend.cpp:66:
FrontEnd API failed with OpConversionFailure: :
Fail to convert elementwise_mul Exception Check 'args_et.is_dynamic() || args_et != element::boolean' failed at path_to_openvino\src\core\src\op\util\binary_elementwise_arithmetic.cpp:32:
While validating node 'opset1::Multiply Multiply_19804 (tmp_253[0]:boolean[?,?,4], tmp_255[0]:boolean[?,?,4]) -> (dynamic[...])' with friendly_name 'Multiply_19804':
Arguments cannot have boolean element type (argument element type: boolean).

[xczhai]

@xuewenyuan
Please file a new issue for this. It is much appreciated if you describe how to export or upload paddle inference model pdmodel.

[avitial]

Closing this as last PR addressing the issue has been merged to master branch.