dec_ref bug with asyncio

[artificial-lshamis]

Pybind11 crashes when running with asyncio. The issue seems to be related to reference counting function objects, but I haven't been able to get more precise than that.

The following code crashes reliable within the first few iterations on my machine. Unfortunately, the same code runs fine on some other machines, even within the same Docker image.

C++ code:

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>

#include <thread>

namespace py = pybind11;

PYBIND11_MODULE(test, m) {
  m.def("do_stuff", [](std::function<void(int)> fn) {
    static int cnt = 0;
    int mycnt = cnt++;
    std::thread t{[fn, mycnt]() {
      std::this_thread::sleep_for(std::chrono::milliseconds(10));
      fn(mycnt);
    }};
    t.detach();
  });
}

Python code:

import asyncio
import time
import test

async def helper():
  loop = asyncio.get_event_loop()
  fut = asyncio.Future(loop=loop)
  def wait_for_reply(reply):
    loop.call_soon_threadsafe(fut.set_result, reply)
  test.do_stuff(wait_for_reply)
  return await fut

async def main():
  while True:
    val = await helper()
    print(val)

asyncio.get_event_loop().set_debug(True)
asyncio.get_event_loop().run_until_complete(main())

Stacktrace:

Thread 2 "python3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff416b700 (LWP 12)]
0x00000000004d610d in ?? ()
(gdb) bt
#0  0x00000000004d610d in ?? ()
#1  0x000000000048c311 in ?? ()
#2  0x00007ffff4e38790 in pybind11::handle::dec_ref() const & (this=0xc8dba0)
    at external/pybind11_archive/include/pybind11/pytypes.h:165
#3  0x00007ffff4e38842 in pybind11::object::~object (this=0xc8dba0, 
    __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/pytypes.h:208
#4  0x00007ffff4e3e0c4 in pybind11::function::~function (this=0xc8dba0, 
    __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/pytypes.h:1226
#5  0x00007ffff4e522be in pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}::~handle() (
    this=0xc8dba0, __in_chrg=<optimized out>)
    at external/pybind11_archive/include/pybind11/functional.h:57
#6  0x00007ffff4e5462e in std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (
    __victim=...) at /usr/include/c++/7/bits/std_function.h:207
#7  0x00007ffff4e53d65 in std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}>::_M_manager(std::_Any_data&, std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (int)>, void>::load(pybind11::handle, bool)::{lambda(int)#1}> const&, std::_Manager_operation) (__dest=..., 
    __source=..., __op=std::__destroy_functor)
    at /usr/include/c++/7/bits/std_function.h:231
#8  0x00007ffff4e42407 in std::_Function_base::~_Function_base (this=0xbe7a58, 
    __in_chrg=<optimized out>) at /usr/include/c++/7/bits/std_function.h:276
#9  0x00007ffff4e4254a in std::function<void (int)>::~function() (
    this=0xbe7a58, __in_chrg=<optimized out>)
    at /usr/include/c++/7/bits/std_function.h:389
#10 0x00007ffff4e36802 in <lambda(std::function<void(int)>)>::<lambda()>::~<lambda>(void) (this=0xbe7a58, __in_chrg=<optimized out>) at test.py.cc:12
#11 0x00007ffff4e36958 in std::_Head_base<0, pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()>, false>::~_Head_base(void) (
    this=0xbe7a58, __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:120
#12 0x00007ffff4e36974 in std::_Tuple_impl<0, pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> >::~_Tuple_impl(void) (
    this=0xbe7a58, __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:343
#13 0x00007ffff4e36990 in std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> >::~tuple(void) (this=0xbe7a58, 
    __in_chrg=<optimized out>) at /usr/include/c++/7/tuple:556
#14 0x00007ffff4e369ac in std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > >::~_Invoker(void) (this=0xbe7a58, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:221
#15 0x00007ffff4e378f8 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > > >::~_State_impl(void) (this=0xbe7a50, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:178
#16 0x00007ffff4e37920 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<pybind11_init_test(pybind11::module&)::<lambda(std::function<void(int)>)>::<lambda()> > > >::~_State_impl(void) (this=0xbe7a50, __in_chrg=<optimized out>)
    at /usr/include/c++/7/thread:178
#17 0x00007ffff448273c in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007ffff7bbd6db in start_thread (arg=0x7ffff416b700)
    at pthread_create.c:463
#19 0x00007ffff6cf288f in clone ()

(Note: This used pybind11 at commit 435dbdd, but fails with other versions as well)

Dockerfile:

FROM ubuntu:bionic

RUN apt-get update -y && apt-get install -y gdb python2.7-dev python3-dev python-pip python3-pip

....

ENTRYPOINT ["gdb", "-ex=r", "--args", "/usr/bin/python3", "test.py"]

(Note: this also fails on ubuntu:xenial)

[Xfel]

When using C++ threads, make sure to aquire python's global interpreter lock before calling a python function.

[virtuald]

Seems related to #1587/#1595

[bstaletic]

This seems to work in 2.6.0. Feel free to reopen if it is still broken.