[Bug]: Cannot convert pytorch YOLOv9 model directly

[sungchul2]

OpenVINO Version

2024.3.0

Operating System

Ubuntu 20.04 (LTS)

Device used for inference

CPU

Framework

PyTorch

Model used

https://github.com/sungchul2/training_extensions/blob/yolov9-integration/src/otx/algo/detection/yolov9.py

Issue description

Error when converting YOLOv9 model to OpenVINO IR.
It seems that fe_output_user_data_repack return the list of dicts that don't have name key.

openvino/tools/ovc/openvino/tools/ovc/moc_frontend/pipeline.py

Lines 134 to 136 in 1e3b88e

	_outputs = fe_output_user_data_repack(input_model, argv.output, moc_front_end.get_name())
	for out_desc in _outputs:
	oplaces.append(out_desc["name"])

openvino/tools/ovc/openvino/tools/ovc/moc_frontend/extractor.py

Lines 308 to 339 in 22079ca

	def fe_output_user_data_repack(input_model: InputModel, outputs: list, framework: str):
	"""
	:param input_model: Input Model to operate on
	:param outputs: list of node names provided by user
	:return: dictionary with node IDs as keys and list of port dictionaries as values
	Example of outputs dictionary:
	_outputs =
	{
	'node_ID':
	[
	{'out': 0},
	{'out': 1},
	],
	'node_1_ID':
	[
	{'port': None}
	],
	'node_2_ID':
	[
	{'in': 3}
	]
	}
	"""
	_outputs = []
	if outputs is not None:
	for output in outputs:
	node = decode_name_with_port(input_model, output, framework, IOType.Output)
	if node is None:
	raise Error("Cannot find location {} in the graph".format(output))
	_outputs.append({"node": node, "output_name": output})
	return _outputs

Step-by-step reproduction

import torch
import openvino
from functools import partial
from otx.algo.detection.yolov9 import YOLOv9

model = YOLOv9(model_name="yolov9_s", label_info=2)
dummy_tensor = torch.rand(1, 3, 640, 640).to(next(model.parameters()).device)

original_forward = model.model.forward
try:
    model.model.forward = partial(model.model.export, batch_img_metas=[], rescale=False)
    exported_model = openvino.convert_model(
        model.model,
        example_input=dummy_tensor,
        input=(openvino.runtime.PartialShape((1, 3, 640, 640)),),
        output=["boxes", "labels"],
    )
finally:
    model.model.forward = original_forward

Relevant log output

...
  File "venv/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 494, in _convert
    ov_model = driver(argv, {"conversion_parameters": non_default_params})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 250, in driver
    ov_model = moc_emit_ir(prepare_ir(argv), argv)
                           ^^^^^^^^^^^^^^^^
  File "venv/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 194, in prepare_ir
    ov_model = moc_pipeline(argv, moc_front_end)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/moc_frontend/pipeline.py", line 136, in moc_pipeline
    oplaces.append(out_desc["name"])
                   ~~~~~~~~^^^^^^^^
KeyError: 'name'

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[sungchul2]

If running without input and output, it's fine.
But if using output to set names to outputs, the error occurred.

exported_model = openvino.convert_model(model.model, example_input=dummy_tensor)  # O
exported_model = openvino.convert_model(model.model, example_input=dummy_tensor, output=["boxes", "labels"])  # X

[sungchul2]

Likewise https://docs.openvino.ai/2024/notebooks/yolov9-optimization-with-output.html#convert-pytorch-model-to-openvino-ir

[mvafin]

@sungchul2 What is the purpose of specifying the names of outputs? Is it renaming of outputs?
As I see the model outputs is tuple which we unpack internally, original model have in fact a single output and your use case will fail, because you specify several outputs.
We could support such case of renaming outputs if the number of outputs after our internal unpacking of tuple match the specified ones, I will plan this for future.

[sungchul2]

@sungchul2 What is the purpose of specifying the names of outputs? Is it renaming of outputs? As I see the model outputs is tuple which we unpack internally, original model have in fact a single output and your use case will fail, because you specify several outputs. We could support such case of renaming outputs if the number of outputs after our internal unpacking of tuple match the specified ones, I will plan this for future.

Yes, I want to change the output names to use model api.
Our use case has 2 outputs, corresponding to bboxes and labels.
And in openvino example, ov_model seems to have 4 outputs:

>>> ov_model
<Model: 'Model0'                                                                                                                                            inputs[
<ConstOutput: names[x] shape[?,3,?,?] type: f32>
]
outputs[
<ConstOutput: names[] shape[?,84,8400] type: f32>,
<ConstOutput: names[xi.1] shape[?,144,4..,4..] type: f32>,
<ConstOutput: names[xi.3] shape[?,144,2..,2..] type: f32>,
<ConstOutput: names[xi] shape[?,144,1..,1..] type: f32>
]>

>>> ov_model.outputs
[<Output: names[] shape[?,84,8400] type: f32>, <Output: names[xi.1] shape[?,144,4..,4..] type: f32>, <Output: names[xi.3] shape[?,144,2..,2..] type: f32>, <Output: names[xi] shape[?,144,1..,1..] type: f32>]

I tried to set names to each output but failed with the same error:

>>> ov_model = ov.convert_model(model, example_input=example_input, output=["boxes", "labels", "a", "b"])                                          [41/1855]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert.py", line 100, in convert_model
    ov_model, _ = _convert(cli_parser, params, True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 548, in _convert
    raise e
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 494, in _convert
    ov_model = driver(argv, {"conversion_parameters": non_default_params})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 250, in driver
    ov_model = moc_emit_ir(prepare_ir(argv), argv)
                           ^^^^^^^^^^^^^^^^
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/convert_impl.py", line 194, in prepare_ir
    ov_model = moc_pipeline(argv, moc_front_end)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sungchul/.pyenv/versions/3.11.8/envs/otx_v2/lib/python3.11/site-packages/openvino/tools/ovc/moc_frontend/pipeline.py", line 136, in moc_pipeli
ne
    oplaces.append(out_desc["name"])
                   ~~~~~~~~^^^^^^^^
KeyError: 'name'

[mvafin]

For pytorch this is a tricky task to figure out what needs to be done when you specify output names, we discussed internally this case and we think that we cannot provide a robust enough solution and it may confuse users.
For your case you can update the name on the resulting model with code like this:

ov_model = ov.convert_model(...)
output_names = ["boxes", "labels"]
for out, out_name in zip(ov_model.outputs, output_names):
    out.get_tensor().set_names({out_name})
ov_model.validate_nodes_and_infer_types()

[sungchul2]

Thanks for your comment. I agree with your team's concern about the difficulty of knowing the output.
But I think it is risky to leave this part with the error.
I'd like to suggest updating this part with a warning or supporting this part in any way.
Anyway, I'll update your suggestion in my case, please ping this thread if you have any updates :)

[mvafin]

Thanks for your comment. I agree with your team's concern about the difficulty of knowing the output. But I think it is risky to leave this part with the error. I'd like to suggest updating this part with a warning or supporting this part in any way. Anyway, I'll update your suggestion in my case, please ping this thread if you have any updates :)

Agree, will update to print a better error message