Embedded python: Pass an C++ object by reference

[singhanubhav2000]

I have a use-case where I have implemented an API in python and that needs to be invoked by C++.

struct ABC {
ABC(int _i, int _j) {
i = _i;
j = _j;
}
int i;
int j;
};

struct listABC {
listABC() {}
std::string name_;
std::vector lst_;
};

PYBIND11_MAKE_OPAQUE(std::vector);
PYBIND11_MODULE(cc11binds, m) {
py::class(m, "ABC")
.def(py::init<double, double>())
;
py::class_(m, "listABC")
.def_readwrite("name", &listABC::name_)
.def_readwrite("lst", &listABC::lst_)
;
py::bind_vector<std::vector>(m, "ABCTest", py::module_local(false));
}

struct listABC { std::string name_; std::vector lst_; };

cumabc(&abcList) -> this takes reference to vector object. This works fine:

before cumabc, in C++ --- 2
[i:1 :: j:2]
after cumabc, in C++ --- 2
[i:1 :: j:2, i:3 :: j:4]
Where (3,4) is added by python interpreter.

cumlistABC(&listAbcWrapper) -> This takes reference to listABC object.

My code encounters both the type of pass by reference. Both listAbcWrapper and abcList are passed by class A to B. I was curious if we need to manage the reference count of listAbcWrapper and abcList (i.e. increase the reference count till their lifetime in A. A can later call a freelistAbc and freeCumAbc).
My architecture is:
|-----------------| |------------------| |--------------------|
| Main Driver A | --> | C++ Wrapper B | -----> | Python Callee C |
|-----------------| |------------------| |--------------------|

[vanossj]

is this different that #1508? Code doesn't compile, need a more complete example.

[singhanubhav2000]

The code is similar but this is a different discussion. All I wanted was how to manage reference count for the scenario I mentioned above.

[vanossj]

If you pass around pybind11::object objects, then the reference counting is handled for you.

If you use the python c api, like raw PyObject, or pybind11::handle, then you need to handle reference counting for those objects manually.

[singhanubhav2000]

I have a problem where the code is panicing when I am making python call from C++. Its on the line of the above discussion:
the python snippet is
import utils
import TplatPythonInterface

class TplatPyModule:
def init(self):
pass

    def connect(self, tplatEndpointInfo):
        self.cls = utils.getClassByName(tplatEndpointInfo.plugin_path,
                                        tplatEndpointInfo.sub_plugin,
                                        tplatEndpointInfo.application)
        self.cls.connect(tplatEndpointInfo.endpoint_ip, tplatEndpointInfo.blob)

Here is binding code
#include "embed.h"
#include "stl_bind.h"
#include "tplat_py_wrapper.h"
#include "sma_interface.h"
namespace py = pybind11;

Here epInfo_(of type tplatEndpointInfo*) is defined in the base class smaTplatInterface,

class attribute((visibility("hidden"))) tplatPyWrapper :
public smaTplatInterface
{
static pybind11::module module_;
pybind11::object pyModule_; <-- Pointer to python class
static pybind11::object import(const std::string& module,
const std::string& path,
pybind11::object& globals);
public:
static void initialize();
static void finalize();
tplatPyWrapper(tplatEndpointInfo* epInfo);
~tplatPyWrapper() {}

SmaStatus connect();

};

py::module tplatPyWrapper::module_;

PYBIND11_MAKE_OPAQUE(tplatEndpointInfo);

PYBIND11_EMBEDDED_MODULE(TplatPythonInterface, m) {
m.doc() = "Thirdplatform framework to python binding";
py::class_(m, "TplatEndpointInfo", py::call_guard())
.def(py::init<>(), py::return_value_policy::reference_internal)
.def_readwrite("application", &tplatEndpointInfo::application_,
py::return_value_policy::reference_internal)
.def_readwrite("endpoint_path", &tplatEndpointInfo::endpoint_path_,
py::return_value_policy::reference_internal)
.def_readwrite("endpoint_ip", &tplatEndpointInfo::endpoint_ip_,
py::return_value_policy::reference_internal)
.def_readwrite("sub_plugin", &tplatEndpointInfo::sub_plugin_,
py::return_value_policy::reference_internal)
.def_readwrite("plugin_path", &tplatEndpointInfo::plugin_path_,
py::return_value_policy::reference_internal)
.def_readwrite("blob", &tplatEndpointInfo::blob_,
py::return_value_policy::reference_internal)

The code is panicing at the time when we are trying to call the connect function as defined under.

static void tplatPyWrapper::initialize() {
py::initialize_interpreter();
py::object main = py::module::import("main");
py::object globals = main.attr("dict");
module_ = py::module::import("tplatPyModule");
}

tplatPyWrapper::tplatPyWrapper() {
auto tplatPyModuleClass = module_.attr("TplatPyModule");
pyModule_ = tplatPyModuleClass();
}

SmaStatus
tplatPyWrapper::connect()
{
auto connect = pyModule_.attr("connect");
connect(epInfo_);
return SMA_OK;
}
initialize();
tplatPyWrapper* tp = new tplatPyWrapper(epInfo_);
tp->connect();

(gdb) bt
#0 0x00007ffff57845f7 in raise () from /lib64/libc.so.6
#1 0x00007ffff5785ce8 in abort () from /lib64/libc.so.6
#2 0x00007ffff60889d5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
#3 0x00007ffff6086946 in ?? () from /lib64/libstdc++.so.6
#4 0x00007ffff6086973 in std::terminate() () from /lib64/libstdc++.so.6
#5 0x00007ffff6086b93 in _cxa_throw () from /lib64/libstdc++.so.6
#6 0x00000000008f32a3 in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call (this=0x7fffef9b4950, ptr=0xdbe690) at ../../../common/pybind11/cast.h:1943
#7 0x00000000008f00ee in pybind11::detail::object_api<pybind11::detail::accessorpybind11::detail::accessor_policies::str_attr >::operator()<(pybind11::return_value_policy)1, tplatEndpointInfo*&> (this=0x7fffef9b4990) at ../../../common/pybind11/cast.h:2096
#8 0x00000000008d97c2 in tplatPyWrapper::connect (this=0x7fffcc01b6c0) at tplat_py_wrapper.cc:130

[singhanubhav2000]

Hi Guys,
Any idea how to fix the issue? Any help would be greatly appreciated.