Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dec_ref bug with asyncio #1521

Closed
artificial-lshamis opened this issue Sep 12, 2018 · 3 comments
Closed

dec_ref bug with asyncio #1521

artificial-lshamis opened this issue Sep 12, 2018 · 3 comments

Comments

@artificial-lshamis
Copy link

Pybind11 crashes when running with asyncio. The issue seems to be related to reference counting function objects, but I haven't been able to get more precise than that.

The following code crashes reliable within the first few iterations on my machine. Unfortunately, the same code runs fine on some other machines, even within the same Docker image.

C++ code:

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>

#include <thread>

namespace py = pybind11;

PYBIND11_MODULE(test, m) {
  m.def("do_stuff", [](std::function<void(int)> fn) {
    static int cnt = 0;
    int mycnt = cnt++;
    std::thread t{[fn, mycnt]() {
      std::this_thread::sleep_for(std::chrono::milliseconds(10));
      fn(mycnt);
    }};
    t.detach();
  });
}

Python code:

import asyncio
import time
import test

async def helper():
  loop = asyncio.get_event_loop()
  fut = asyncio.Future(loop=loop)
  def wait_for_reply(reply):
    loop.call_soon_threadsafe(fut.set_result, reply)
  test.do_stuff(wait_for_reply)
  return await fut

async def main():
  while True:
    val = await helper()
    print(val)

asyncio.get_event_loop().set_debug(True)
asyncio.get_event_loop().run_until_complete(main())

Stacktrace:

Thread 2 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff416b700 (LWP 12)]
0x00000000004d610d in ?? ()
(gdb) bt
#0  0x00000000004d610d in ?? ()
#1  0x000000000048c311 in ?? ()
#2  0x00007ffff4e38790 in pybind11::handle::dec_ref() const & (this=0xc8dba0)
    at external/pybind11_archive/include/pybind11/pytypes.h:165
#3  0x00007ffff4e38842 in pybind11::object::~object (this=0xc8dba0, 
    __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/pytypes.h:208
#4  0x00007ffff4e3e0c4 in pybind11::function::~function (this=0xc8dba0, 
    __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/pytypes.h:1226
#5  0x00007ffff4e522be in pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}::~handle() (
    this=0xc8dba0, __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/functional.h:57
#6  0x00007ffff4e5462e in std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (
    __victim=...) at /usr/include/c++/7/bits/std_function.h:207
#7  0x00007ffff4e53d65 in std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}>::_M_manager(std::_Any_data&, std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}> const&, std::_Manager_operation) (__dest=..., 
    __source=..., __op=std::__destroy_functor)
    at /usr/include/c++/7/bits/std_function.h:231
#8  0x00007ffff4e42407 in std::_Function_base::~_Function_base (this=0xbe7a58, 
    __in_chrg=<optimized out>) at /usr/include/c++/7/bits/std_function.h:276
#9  0x00007ffff4e4254a in std::function<void (int)>::~function() (
    this=0xbe7a58, __in_chrg=<optimized out>)
    at /usr/include/c++/7/bits/std_function.h:389
#10 0x00007ffff4e36802 in <lambda(std::function<void(int)>)>::<lambda()>::~<lambda>(void) (this=0xbe7a58, __in_chrg=<optimized out>) at test.py.cc:12
#11 0x00007ffff4e36958 in std::_Head_base<0, pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()>, false>::~_Head_base(void) (
    this=0xbe7a58, __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:120
#12 0x00007ffff4e36974 in std::_Tuple_impl<0, pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> >::~_Tuple_impl(void) (
    this=0xbe7a58, __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:343
#13 0x00007ffff4e36990 in std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> >::~tuple(void) (this=0xbe7a58, 
    __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:556
#14 0x00007ffff4e369ac in std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > >::~_Invoker(void) (this=0xbe7a58, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:221
#15 0x00007ffff4e378f8 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > > >::~_State_impl(void) (this=0xbe7a50, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:178
#16 0x00007ffff4e37920 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > > >::~_State_impl(void) (this=0xbe7a50, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:178
#17 0x00007ffff448273c in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007ffff7bbd6db in start_thread (arg=0x7ffff416b700)
    at pthread_create.c:463
#19 0x00007ffff6cf288f in clone ()

(Note: This used pybind11 at commit 435dbdd, but fails with other versions as well)

Dockerfile:

FROM ubuntu:bionic

RUN apt-get update -y && apt-get install -y gdb python2.7-dev python3-dev python-pip python3-pip

....

ENTRYPOINT ["gdb", "-ex=r", "--args", "/usr/bin/python3", "test.py"]

(Note: this also fails on ubuntu:xenial)

@Xfel
Copy link

Xfel commented Sep 16, 2018

When using C++ threads, make sure to aquire python's global interpreter lock before calling a python function.

@virtuald
Copy link
Contributor

virtuald commented Jan 1, 2019

Seems related to #1587/#1595

@bstaletic
Copy link
Collaborator

This seems to work in 2.6.0. Feel free to reopen if it is still broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants