Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embedding python in C++ Code to pass stl containers by reference #1508

Closed
singhanubhav2000 opened this issue Aug 31, 2018 · 10 comments
Closed

Comments

@singhanubhav2000
Copy link

I have a usecase where I have implemented an API in python and it needs to return struct, vector or map to C++. Could you please help me with an example how to do that. All the usecases I see in pybind is just simply python being embedded in C++ without any return value. The scenario which I need to implement is a python -> C++ interface where the C++ caller will start a session and call various python objects when it wants and then close the session. Any help would be highly appreciated.

I tried to implement the solution as given under the link (http://zpz.github.io/embedding-python-in-cpp-3) scenario 3 but that is failing for me with the error:
[root@scspa0521148001 py4cc]# ./main
=== pass as &x ===

before cumsum, in C++ --- [1, 3, 5]
terminate called after throwing an instance of 'pybind11::cast_error'
what(): make_tuple(): unable to convert argument of type 'std::vector<int, std::allocator >*' to Python object
Aborted (core dumped)

py4cc.zip

@vanossj
Copy link
Contributor

vanossj commented Sep 6, 2018

For test_ref.cc, use the pybind embedded interpreter when using pybind11 modules as per the docs warning. Just replace two lines.

@@ -1,4 +1,4 @@
-#include "Python.h"
+#include <pybind11/embed.h> // everything needed for embedding
 #include "pybind11/pybind11.h"
 
 #include "util.h"
@@ -100,7 +100,7 @@ void test_ref()
     
 int main()
 {
-	Py_Initialize();
+	py::scoped_interpreter guard{}; // start the interpreter and keep it alive
 	test_ref();
 	return 0;
 }

For _c11binds.cc, you need to add two things described in the docs.
First add PYBIND11_MAKE_OPAQUE macros for each bound stl type, otherwise these types will never be passed by reference.
Additionally, because binding stl types is common, these bindings are keep hidden (unless they involve a custom type somehow). If you imported two modules that bound std::vector<int> to different python types, your calling code wouldn't be able to resolve how to cast that object. To force these bindings to be exposed you have to add py::module_local(false) as an argument to your py::bind* calls.

Had you bound a vector of your custom class, those bindings would be visible globally by default. py::bind_vector<std::vector<ABC>>(m, "ABCVector");

I also changed to the newer PYBIND11_MODULE syntax

@@ -6,16 +6,21 @@
 
 namespace py = pybind11;
 
+PYBIND11_MAKE_OPAQUE(std::vector<int>);
+PYBIND11_MAKE_OPAQUE(std::vector<std::string>);
+typedef std::map<std::string,double> stringdoublemap;
+PYBIND11_MAKE_OPAQUE(stringdoublemap); // because PYBIND11_MAKE_OPAQUE(std::map<std::string,double>) was not compiling for me
+
 struct ABC {
 	int i;
 	int j;
 };
 
-PYBIND11_PLUGIN(_cc11binds) {
-    py::module m("_cc11binds", "C++ type bindings created by py11bind");
-    py::bind_vector<std::vector<int>>(m, "IntVector");
-    py::bind_vector<std::vector<std::string>>(m, "StringVector");
-    py::bind_map<std::map<std::string, double>>(m, "StringDoubleMap");
+
+PYBIND11_MODULE(_cc11binds, m) {
+    m.doc() = "C++ type bindings created by py11bind";
+    py::bind_vector<std::vector<int>>(m, "IntVector", py::module_local(false));
+    py::bind_vector<std::vector<std::string>>(m, "StringVector", py::module_local(false));
+    py::bind_map<std::map<std::string, double>>(m, "StringDoubleMap", py::module_local(false));
     // 'bind_map` does not create method `values`.
-    return m.ptr();
 }

@bstaletic
Copy link
Collaborator

PYBIND11_MAKE_OPAQUE(std::mapstd::string,double) was not compiling

That's because PYBIND11_MAKE_OPAQUE() is a macro. Macros don't understand constructs as foo<bar, baz> and treat them as two arguments - foo<bar and baz>.

@vanossj
Copy link
Contributor

vanossj commented Sep 6, 2018

But there is a test for that

@singhanubhav2000
Copy link
Author

singhanubhav2000 commented Sep 17, 2018

Thank you this code is working for me. I made some changes to the code where in I wanted to pass a reference struct to embedded python function.
struct listABC { std::string name_; std::vector<ABC> lst_; };

I added the following function in stl.py
from _cc11binds import listABC
def cumlistABC(a): # cumsumfor sequencexwhose elements have meaningful+ operation. y = ABC(3,4) x.append(y) a.lst_.append(a) a.name_ = "xyz"

Now when I am trying to invoke this function in C++
listABC listAbc;
cumlistABC(&listAbc);

It throws an error "unable to convert argument of type 'listABC*' to Python object"
I am trying to understand how to pass listAbc by reference so that the caller C++ can see the changes made in python.

@bstaletic
Copy link
Collaborator

Looks like you didn't bind listABC.

@singhanubhav2000
Copy link
Author

singhanubhav2000 commented Sep 17, 2018

Ohhh sorry missed that line in my previous comment.
PYBIND11_MAKE_OPAQUE(std::vector<ABC>);
PYBIND11_MODULE(_cc11binds, m) {
py::class_<ABC>(m, "ABC")
.def(py::init<double, double>())
;
py::class_<listABC>(m, "listABC")
.def_readwrite("name", &listABC::name_)
.def_readwrite("lst", &listABC::lst_)
;
py::bind_vector<std::vector<ABC>>(m, "ABCTest", py::module_local(false));
}

Adding the full code for understanding. By the way I have removed _cc11binds dependency so that entire c++ code is in one file. What I am confused about is passing vector to python is fine.

before cumabc, in C++ --- 2
[i:1 :: j:2]
after cumabc, in C++ --- 2
[i:1 :: j:2, i:3 :: j:4]
Where (3,4) is added by python interpreter.

But the moment, we pass object of type listABC as a reference to python interpreter,

struct listABC { std::string name_; std::vector<ABC> lst_; };

I don't see any updates done by python interpreter to c++ caller i.e. it is passed by value.

test.zip

@vanossj
Copy link
Contributor

vanossj commented Sep 21, 2018

the problem is cumlistABC(py::cast(listAbc));

py::cast(listAbc) will create a python object with a copy of listAbc. To have python affect the c++ object, cast the pointer cumlistABC(py::cast(&listAbc)); or just pass the pointer and pybind11 will do the cast for you cumlistABC(&listAbc);

@singhanubhav2000
Copy link
Author

I have another doubt here. I was trying to make the program multithreaded.
But I am getting a panic.
All I did was starting multiple threads and doing gil_acquire & gil_release before and after each and every python call. Do we need to do that or not? Attaching the code.

#0 0x00007ffff7a98ce0 in _PyTrash_thread_destroy_chain () from /lib64/libpython2.7.so.1.0
#1 0x00007ffff7a5b39a in PyObject_CallFunctionObjArgs () from /lib64/libpython2.7.so.1.0
#2 0x00007ffff7a5bb7f in PyObject_IsSubclass () from /lib64/libpython2.7.so.1.0
#3 0x00007ffff7a6ac1e in instancemethod_descr_get () from /lib64/libpython2.7.so.1.0
#4 0x00007ffff7ab1a1e in lookup_method () from /lib64/libpython2.7.so.1.0
#5 0x00007ffff7ab1a71 in slot_tp_init () from /lib64/libpython2.7.so.1.0
#6 0x00007ffff7ab079f in type_call () from /lib64/libpython2.7.so.1.0
#7 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#8 0x00007ffff7aef236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a7f978 in function_call () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7aec8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#13 0x000000000042012f in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call (this=0x7ffff02e2bb0, ptr=0x7ffff7e9ab90) at /root/pybind11/include/pybind11/cast.h:1941
#14 0x000000000041cf42 in pybind11::detail::object_api<pybind11::detail::accessorpybind11::detail::accessor_policies::str_attr >::operator()<(pybind11::return_value_policy)1, std::vector<ABC, std::allocator >>(std::vector<ABC, std::allocator >&&) const (this=0x7ffff02e2c40) at /root/pybind11/include/pybind11/cast.h:2096
#15 0x0000000000406aca in testing_ref (id=1) at test_ref.cc:96
#16 0x0000000000477fd8 in std::_Bind_simple<void ((int))(int)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x774250) at /usr/include/c++/4.8.2/functional:1732
#17 0x0000000000477ee5 in std::_Bind_simple<void (
(int))(int)>::operator()() (this=0x774250) at /usr/include/c++/4.8.2/functional:1720
#18 0x0000000000477e7e in std::thread::_Impl<std::_Bind_simple<void (*(int))(int)> >::_M_run() (this=0x774238) at /usr/include/c++/4.8.2/thread:115
#19 0x00007ffff77bc2a0 in ?? () from /lib64/libstdc++.so.6
#20 0x00007ffff6c19dc5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007ffff6f241cd in clone () from /lib64/libc.so.6

Attaching the changed code.
test.zip

@vanossj
Copy link
Contributor

vanossj commented Sep 23, 2018

What if you use the pybind11 functions for GIL control?

You have to be extra careful when using the embedded interpreter and treading. You can't create multiple interpreters, so you would have to create threads of testing_ref

Another thing I noticed when i was running your previous code, you have py::module mod; declared at the global level. This was throwing address bound errors when I closed the program. If you move the declaration into test_ref() then pass it onto testing_ref(), then mod will be created for the first time after thy embedded interpreter has been initialized and I didn't have any more errors when closing the program.

void test_ref()
{
	py::module mod = py::module::import("stl");
	testing_ref(2, mod);
}

@singhanubhav2000
Copy link
Author

I think I got the problem. I was unnecessarily taking GIL lock before making the python call on the lines of boost::python. Will mark this issue as closed. Thanks for the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants