[Good First Issue][TF FE]: Support complex tensors for Inv operation

[rkazants]

Context

OpenVINO component responsible for support of TensorFlow models is called as TensorFlow Frontend (TF FE). TF FE converts a model represented in TensorFlow opset to a model in OpenVINO opset.
Some audio models use tensors of complex type. Complex type tensor is a tensor that has elements of complex type. For example, 1D tensor with three elements x = [1+2j, 2, -2j].

For supporting Inv operation on complex type tensor, you need to extend the corresponding loader for Inv.

What needs to be done?

The existing loader for Inv needs to be extended by propagating ComplexTypeMark from input to output and to represent output complex type tensor as a floating-point type tensor with auxiliary dimension that concatenates real and imaginary parts of complex tensor.
To validate the extension, the corresponding layer test needs to be updated with complex tensor cases.

Here is an example of how to extend Reshape loader to support complex type tensors:

OutputVector translate_reshape_op(const NodeContext& node) {
    default_op_checks(node, 2, {"Reshape"}, true);
    auto tensor = node.get_input(0);
    auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());
    auto shape = node.get_input(1);
    if (complex_type_mark) {
        element::Type complex_part_type = complex_type_mark->get_complex_part_type();
        tensor = complex_type_mark->input_value(0);

        OutputVector concat_inputs;
        concat_inputs.push_back(shape);
        concat_inputs.push_back(make_shared<v0::Constant>(shape.get_element_type(), Shape{1}, 2));

        auto concat = make_shared<v0::Concat>(concat_inputs, 0);
        auto reshape = make_shared<v1::Reshape>(tensor, concat, false);
        set_node_name(node.get_name(), reshape);
        auto complex_reshape = make_shared<ComplexTypeMark>(reshape, complex_part_type);
        return {complex_reshape->output(0)};
    }

    auto reshape = make_shared<v1::Reshape>(tensor, shape, false);
    set_node_name(node.get_name(), reshape);
    return {reshape};
}

Since OpenVINO does not have native support of complex tensors, we handle complex type in intermediate layers by representing them as a floating-point type with additional dimension (specially created) to store real and imaginary parts of the original complex tensor so slicing by the last dimension will give either real or imaginary parts: x[...,0] - real and x[...,1] - imaginary parts.

On the first step, we update default_op_checks with true flag to indicate that loader for Reshape operation now handles complex tensors:

default_op_checks(node, 2, {"Reshape"}, true);

Secondly, we check if complex type mark exists by anticipated inputs. This mark indicates that input tensor of complex type:

auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());

Thirdly, we retrieve a floating-point tensor (with additional dimension to store real and imaginary parts) simulating complex tensor:

tensor = complex_type_mark->input_value(0);

After that, we implement conversion for Reshape for this particular case. Since a floating-point tensor simulating complex tensor has additional dimension equal to 2,
we update input target shape by appending 2 value and perform reshape on a floating-point tensor simulating complex tensor.

Finally, since Reshape should produce complex tensor by output we insert a new mark ComplexTypeMark into the output.

To validate support of complex tensors for Reshape, the new layer test TestComplexReshape was added.

Example how to run the layer test:

export TEST_DEVICE=CPU
cd openvino/tests/layer_tests/tensorflow_tests
pytest test_tf_Reshape.py

Example Pull Requests

• Extensions for Shape, Mul, Reshape in [TF FE] Complex type support extended for Separate Bass model. #21477
• Extension for Roll operation in [TF FE] Support Complex Tensors #20860

Resources

• What is OpenVINO?
• How to Build OpenVINO
• Developer documentation for TensorFlow Frontend
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers

Contact points

• @openvinotoolkit/openvino-tf-frontend-maintainers
• rkazants in Discord

Ticket

No response

[linzs148]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[linzs148]

@rkazants Hello, I have compiled openvino with

cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_PYTHON=ON -DENABLE_TESTS=ON -DENABLE_FUNCTIONAL_TESTS=ON -DENABLE_DEBUG_CAPS=ON -DENABLE_CPU_DEBUG_CAPS=ON -DENABLE_NCC_STYLE=ON ..

and set environment variables

export PYTHONPATH=openvino/bin/intel64/Debug/python:$PYTHONPATH
export LD_LIBRARY_PATH=/root/openvino/bin/intel64/Debug:$LD_LIBRARY_PATH

when I run test_tf_Reshape.py

pytest test_tf_Reshape.py

Something wrong happend:

____________________________ ERROR collecting tensorflow_tests/test_tf_Reshape.py _____________________________
ImportError while importing test module '/root/openvino/tests/layer_tests/tensorflow_tests/test_tf_Reshape.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
test_tf_Reshape.py:7: in
from common.tf_layer_test_class import CommonTFLayerTest
../common/tf_layer_test_class.py:5: in
from common.utils.tf_utils import summarize_graph
../common/utils/tf_utils.py:10: in
from openvino.tools.mo.ops.op import PermuteAttrs
E ModuleNotFoundError: No module named 'openvino.tools.mo'

What should I do to fix this?

[rkazants]

Hi @linzs148, this PR should help to resolve the issue: #23067

[linzs148]

Hi @linzs148, this PR should help to resolve the issue: #23067

@rkazants This helps solve the problem above, but there's another problem:

def generate_ir_python_api(coverage=False, **kwargs):
out_dir = kwargs['output_dir'] + os.sep + "model.xml"
if 'use_legacy_frontend' in kwargs and kwargs['use_legacy_frontend']:
from openvino.runtime import serialize
from openvino.tools.mo import convert_model
ov_model = convert_model(**kwargs)
serialize(ov_model, out_dir)
else:
> from openvino import convert_model, save_model
E ImportError: cannot import name 'convert_model' from 'openvino' (/root/openvino/bin/intel64/Debug/python/openvino/init.py)

convert_model depends on openvino.tools:

try:
# Model Conversion API - ovc should reside in the main namespace
from openvino.tools.ovc import convert_model
except ImportError:
pass

[rkazants]

Hi @linzs148,

For running tests you should build your openvino wheel that you will test using the layer tests and install it. Check out this documentation: https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux.md.
Specify -DENABLE_WHEEL=ON for cmake command.

make -j 8 ie_wheel

[linzs148]

Hi @linzs148,

For running tests you should build your openvino wheel that you will test using the layer tests and install it. Check out this documentation: https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux.md. Specify -DENABLE_WHEEL=ON for cmake command.

make -j 8 ie_wheel

Thanks! This makes sense for me.

By the way, my Linux server doesn't have an intel GPU so only tests for CPU can pass. What should I do to test in GPU?

[linzs148]

@rkazants Hi, I added some tests for complex inv and got an error while running pytest:

file /root/openvino/tests/layer_tests/tensorflow_tests/test_tf_Inv.py, line 81
@pytest.mark.parametrize("params", test_data_basic)
@pytest.mark.precommit_tf_fe
@pytest.mark.nightly
def test_complex_inv(self, params, ie_device, precision, ir_version, temp_dir,
E fixture 'use_new_frontend' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, ir_version, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, rename_tf_fe_libs, temp_dir, tflite, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, use_legacy_frontend
> use 'pytest --fixtures [testpath]' for help on them.

What should I do to fix this?

[rkazants]

Hi @linzs148,
For running tests you should build your openvino wheel that you will test using the layer tests and install it. Check out this documentation: https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux.md. Specify -DENABLE_WHEEL=ON for cmake command.

make -j 8 ie_wheel

Thanks! This makes sense for me.

By the way, my Linux server doesn't have an intel GPU so only tests for CPU can pass. What should I do to test in GPU?

Just launch the tests only on CPU.

export TEST_DEVICE=CPU

[rkazants]

@rkazants Hi, I added some tests for complex inv and got an error while running pytest:

file /root/openvino/tests/layer_tests/tensorflow_tests/test_tf_Inv.py, line 81
@pytest.mark.parametrize("params", test_data_basic)
@pytest.mark.precommit_tf_fe
@pytest.mark.nightly
def test_complex_inv(self, params, ie_device, precision, ir_version, temp_dir,
E fixture 'use_new_frontend' not found

  available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, ir_version, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, rename_tf_fe_libs, temp_dir, tflite, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, use_legacy_frontend
  use 'pytest --fixtures [testpath]' for help on them.

What should I do to fix this?

Please fix your test and align with other layer tests. We recently removed use_new_frontend option

[linzs148]

@rkazants Hi, I have modified and compiled inv.cpp but the test result shows that internal error: translator for Inv does not support input complex type

self = <FrontEnd 'tf'>, model = <openvino._pyopenvino.InputModel object at 0x7f0f92209c60>
def convert(self, model: Union[Model, InputModel]) -> Model:
> converted_model = super().convert(model)
E openvino._pyopenvino.OpConversionFailure: Check 'is_conversion_successful' failed at src/frontends/tensorflow/src/frontend.cpp:434:
E FrontEnd API failed with OpConversionFailure:
E [TensorFlow Frontend] Internal error, conversion is failed for Imag operation with a message:
E Check 'complex_type_mark' failed at src/frontends/tensorflow_common/src/op/real_imag.cpp:33:
E FrontEnd API failed with OpValidationFailureWhile validating node 'Imag':
E [TensorFlow Frontend] internal error: ComplexTypeMark is not set at the input of Imag
E [TensorFlow Frontend] Internal error, conversion is failed for Real operation with a message:
E Check 'complex_type_mark' failed at src/frontends/tensorflow_common/src/op/real_imag.cpp:33:
E FrontEnd API failed with OpValidationFailureWhile validating node 'Real':
E [TensorFlow Frontend] internal error: ComplexTypeMark is not set at the input of Real
E [TensorFlow Frontend] Internal error, conversion is failed for Inv operation with a message:
E Check '!has_input_complex_type || supported_complex' failed at src/frontends/tensorflow_common/src/utils.cpp:282:
E FrontEnd API failed with OpValidationFailureWhile validating node 'Inv':
E [TensorFlow Frontend] internal error: translator for Inv does not support input complex type
E [TensorFlow Frontend] Internal error, no translator found for operation(s): ComplexTypeMark
E To facilitate the conversion of unsupported operations, refer to Frontend Extension documentation: https://docs.openvino.ai/latest/openvino_docs_Extensibility_UG_Frontend_Extensions.html
/usr/local/lib/python3.10/dist-packages/openvino/frontend/frontend.py:18: OpConversionFailure

Should I reinstall openvino with wheel or do something else?

PS: I have changed default_op_checks(node, 1, {"Inv"}); to default_op_checks(node, 1, {"Inv"}, true);

[rkazants]

Hi @linzs148, please post PR I will take a look. It can be a problem with the layer test and also with update of the translator.

BTW, are you participating in GSoC? Or just performing it as GFI?

Best regards,
Roman

[linzs148]

@rkazants here is my pr #23085
I am going to participate in GSoC

[linzs148]

@rkazants Hi, what should I do next for this issue?