Embedding python in C++ Code to pass stl containers by reference

[singhanubhav2000]

I have a usecase where I have implemented an API in python and it needs to return struct, vector or map to C++. Could you please help me with an example how to do that. All the usecases I see in pybind is just simply python being embedded in C++ without any return value. The scenario which I need to implement is a python -> C++ interface where the C++ caller will start a session and call various python objects when it wants and then close the session. Any help would be highly appreciated.

I tried to implement the solution as given under the link (http://zpz.github.io/embedding-python-in-cpp-3) scenario 3 but that is failing for me with the error:
[root@scspa0521148001 py4cc]# ./main
=== pass as &x ===

before cumsum, in C++ --- [1, 3, 5]
terminate called after throwing an instance of 'pybind11::cast_error'
what(): make_tuple(): unable to convert argument of type 'std::vector<int, std::allocator >*' to Python object
Aborted (core dumped)

py4cc.zip

[vanossj]

For test_ref.cc, use the pybind embedded interpreter when using pybind11 modules as per the docs warning. Just replace two lines.

@@ -1,4 +1,4 @@
-#include "Python.h"
+#include <pybind11/embed.h> // everything needed for embedding
 #include "pybind11/pybind11.h"
 
 #include "util.h"
@@ -100,7 +100,7 @@ void test_ref()
     
 int main()
 {
-	Py_Initialize();
+	py::scoped_interpreter guard{}; // start the interpreter and keep it alive
 	test_ref();
 	return 0;
 }

For _c11binds.cc, you need to add two things described in the docs.
First add PYBIND11_MAKE_OPAQUE macros for each bound stl type, otherwise these types will never be passed by reference.
Additionally, because binding stl types is common, these bindings are keep hidden (unless they involve a custom type somehow). If you imported two modules that bound std::vector<int> to different python types, your calling code wouldn't be able to resolve how to cast that object. To force these bindings to be exposed you have to add py::module_local(false) as an argument to your py::bind* calls.

Had you bound a vector of your custom class, those bindings would be visible globally by default. py::bind_vector<std::vector<ABC>>(m, "ABCVector");

I also changed to the newer PYBIND11_MODULE syntax

@@ -6,16 +6,21 @@
 
 namespace py = pybind11;
 
+PYBIND11_MAKE_OPAQUE(std::vector<int>);
+PYBIND11_MAKE_OPAQUE(std::vector<std::string>);
+typedef std::map<std::string,double> stringdoublemap;
+PYBIND11_MAKE_OPAQUE(stringdoublemap); // because PYBIND11_MAKE_OPAQUE(std::map<std::string,double>) was not compiling for me
+
 struct ABC {
 	int i;
 	int j;
 };
 
-PYBIND11_PLUGIN(_cc11binds) {
-    py::module m("_cc11binds", "C++ type bindings created by py11bind");
-    py::bind_vector<std::vector<int>>(m, "IntVector");
-    py::bind_vector<std::vector<std::string>>(m, "StringVector");
-    py::bind_map<std::map<std::string, double>>(m, "StringDoubleMap");
+
+PYBIND11_MODULE(_cc11binds, m) {
+    m.doc() = "C++ type bindings created by py11bind";
+    py::bind_vector<std::vector<int>>(m, "IntVector", py::module_local(false));
+    py::bind_vector<std::vector<std::string>>(m, "StringVector", py::module_local(false));
+    py::bind_map<std::map<std::string, double>>(m, "StringDoubleMap", py::module_local(false));
     // 'bind_map` does not create method `values`.
-    return m.ptr();
 }

[bstaletic]

PYBIND11_MAKE_OPAQUE(std::mapstd::string,double) was not compiling

That's because PYBIND11_MAKE_OPAQUE() is a macro. Macros don't understand constructs as foo<bar, baz> and treat them as two arguments - foo<bar and baz>.

[vanossj]

But there is a test for that

[singhanubhav2000]

Thank you this code is working for me. I made some changes to the code where in I wanted to pass a reference struct to embedded python function.
struct listABC { std::string name_; std::vector<ABC> lst_; };

I added the following function in stl.py
from _cc11binds import listABC
def cumlistABC(a): # cumsumfor sequencexwhose elements have meaningful+ operation. y = ABC(3,4) x.append(y) a.lst_.append(a) a.name_ = "xyz"

Now when I am trying to invoke this function in C++
listABC listAbc;
cumlistABC(&listAbc);

It throws an error "unable to convert argument of type 'listABC*' to Python object"
I am trying to understand how to pass listAbc by reference so that the caller C++ can see the changes made in python.

[bstaletic]

Looks like you didn't bind listABC.

[singhanubhav2000]

Ohhh sorry missed that line in my previous comment.
PYBIND11_MAKE_OPAQUE(std::vector<ABC>);
PYBIND11_MODULE(_cc11binds, m) {
py::class_<ABC>(m, "ABC")
.def(py::init<double, double>())
;
py::class_<listABC>(m, "listABC")
.def_readwrite("name", &listABC::name_)
.def_readwrite("lst", &listABC::lst_)
;
py::bind_vector<std::vector<ABC>>(m, "ABCTest", py::module_local(false));
}

Adding the full code for understanding. By the way I have removed _cc11binds dependency so that entire c++ code is in one file. What I am confused about is passing vector to python is fine.

before cumabc, in C++ --- 2
[i:1 :: j:2]
after cumabc, in C++ --- 2
[i:1 :: j:2, i:3 :: j:4]
Where (3,4) is added by python interpreter.

But the moment, we pass object of type listABC as a reference to python interpreter,

struct listABC { std::string name_; std::vector<ABC> lst_; };

I don't see any updates done by python interpreter to c++ caller i.e. it is passed by value.

test.zip

[vanossj]

the problem is cumlistABC(py::cast(listAbc));

py::cast(listAbc) will create a python object with a copy of listAbc. To have python affect the c++ object, cast the pointer cumlistABC(py::cast(&listAbc)); or just pass the pointer and pybind11 will do the cast for you cumlistABC(&listAbc);

[singhanubhav2000]

I have another doubt here. I was trying to make the program multithreaded.
But I am getting a panic.
All I did was starting multiple threads and doing gil_acquire & gil_release before and after each and every python call. Do we need to do that or not? Attaching the code.

#0 0x00007ffff7a98ce0 in _PyTrash_thread_destroy_chain () from /lib64/libpython2.7.so.1.0
#1 0x00007ffff7a5b39a in PyObject_CallFunctionObjArgs () from /lib64/libpython2.7.so.1.0
#2 0x00007ffff7a5bb7f in PyObject_IsSubclass () from /lib64/libpython2.7.so.1.0
#3 0x00007ffff7a6ac1e in instancemethod_descr_get () from /lib64/libpython2.7.so.1.0
#4 0x00007ffff7ab1a1e in lookup_method () from /lib64/libpython2.7.so.1.0
#5 0x00007ffff7ab1a71 in slot_tp_init () from /lib64/libpython2.7.so.1.0
#6 0x00007ffff7ab079f in type_call () from /lib64/libpython2.7.so.1.0
#7 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#8 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7aec8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#13 0x000000000042012f in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call (this=0x7ffff02e2bb0, ptr=0x7ffff7e9ab90) at /root/pybind11/include/pybind11/cast.h:1941
#14 0x000000000041cf42 in pybind11::detail::object_api<pybind11::detail::accessorpybind11::detail::accessor_policies::str_attr >::operator()<(pybind11::return_value_policy)1, std::vector<ABC, std::allocator >>(std::vector<ABC, std::allocator >&&) const (this=0x7ffff02e2c40) at /root/pybind11/include/pybind11/cast.h:2096
#15 0x0000000000406aca in testing_ref (id=1) at test_ref.cc:96
#16 0x0000000000477fd8 in std::_Bind_simple<void ((int))(int)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x774250) at /usr/include/c++/4.8.2/functional:1732
#17 0x0000000000477ee5 in std::_Bind_simple<void ((int))(int)>::operator()() (this=0x774250) at /usr/include/c++/4.8.2/functional:1720
#18 0x0000000000477e7e in std::thread::_Impl<std::_Bind_simple<void (*(int))(int)> >::_M_run() (this=0x774238) at /usr/include/c++/4.8.2/thread:115
#19 0x00007ffff77bc2a0 in ?? () from /lib64/libstdc++.so.6
#20 0x00007ffff6c19dc5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007ffff6f241cd in clone () from /lib64/libc.so.6

Attaching the changed code.
test.zip

[vanossj]

What if you use the pybind11 functions for GIL control?

You have to be extra careful when using the embedded interpreter and treading. You can't create multiple interpreters, so you would have to create threads of testing_ref

Another thing I noticed when i was running your previous code, you have py::module mod; declared at the global level. This was throwing address bound errors when I closed the program. If you move the declaration into test_ref() then pass it onto testing_ref(), then mod will be created for the first time after thy embedded interpreter has been initialized and I didn't have any more errors when closing the program.

void test_ref()
{
	py::module mod = py::module::import("stl");
	testing_ref(2, mod);
}

[singhanubhav2000]

I think I got the problem. I was unnecessarily taking GIL lock before making the python call on the lines of boost::python. Will mark this issue as closed. Thanks for the help.