[Python] SEGFAULT in test_udf_via_substrait when run under CPython debug build

[lysnikolaou]

Describe the bug, including details regarding any error messages, version, and platform.

Hey everyone! 👋

I'm trying to build arrow and PyArrow with a debug build of CPython and run the test suite, but I keep running into a segmentation fault. The crash happens in test_udf_via_substrait here, and I can reproduce it using both 3.13.0b4+ and 3.12.4.

The source of the segmentation fault is the Py_INCREF that's happening here. Because this is under a debug build, Py_INCREF tries to access the thread state to increase the aggregate reference count.

Stripped stack trace to `Py_INCREF` call

libpython3.13td.dylib!reftotal_add (/Users/user/.pyenv/sources/3.13t-dev-debug/Python-3.13-dev/Objects/object.c:84)
libpython3.13td.dylib!_Py_INCREF_IncRefTotal (/Users/user/.pyenv/sources/3.13t-dev-debug/Python-3.13-dev/Objects/object.c:231)
libarrow_python.dylib!Py_INCREF(_object*) (/Users/user/.pyenv/versions/3.13t-dev-debug/include/python3.13td/object.h:835)
libarrow_python.dylib!arrow::py::(anonymous namespace)::PythonUdfKernelState::PythonUdfKernelState(std::__1::shared_ptr<arrow::py::OwnedRefNoGIL>) (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/udf.cc:48)
libarrow_python.dylib!arrow::py::(anonymous namespace)::PythonUdfKernelState::PythonUdfKernelState(std::__1::shared_ptr<arrow::py::OwnedRefNoGIL>) (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/udf.cc:47)
libarrow_python.dylib!std::__1::__unique_if<arrow::py::(anonymous namespace)::PythonUdfKernelState>::__unique_single std::__1::make_unique[abi:ue170006]<arrow::py::(anonymous namespace)::PythonUdfKernelState, std::__1::shared_ptr<arrow::py::OwnedRefNoGIL>&>(std::__1::shared_ptr<arrow::py::OwnedRefNoGIL>&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__memory/unique_ptr.h:689)
libarrow_python.dylib!arrow::py::(anonymous namespace)::PythonUdfKernelInit::operator()(arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&) (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/udf.cc:78)
libarrow_python.dylib!decltype(std::declval<arrow::py::(anonymous namespace)::PythonUdfKernelInit&>()(std::declval<arrow::compute::KernelContext*>(), std::declval<arrow::compute::KernelInitArgs const&>())) std::__1::__invoke[abi:ue170006]<arrow::py::(anonymous namespace)::PythonUdfKernelInit&, arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&>(arrow::py::(anonymous namespace)::PythonUdfKernelInit&, arrow::compute::KernelContext*&&, arrow::compute::KernelInitArgs const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__type_traits/invoke.h:340)
libarrow_python.dylib!arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>> std::__1::__invoke_void_return_wrapper<arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>>, false>::__call[abi:ue170006]<arrow::py::(anonymous namespace)::PythonUdfKernelInit&, arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&>(arrow::py::(anonymous namespace)::PythonUdfKernelInit&, arrow::compute::KernelContext*&&, arrow::compute::KernelInitArgs const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__type_traits/invoke.h:407)
libarrow_python.dylib!std::__1::__function::__alloc_func<arrow::py::(anonymous namespace)::PythonUdfKernelInit, std::__1::allocator<arrow::py::(anonymous namespace)::PythonUdfKernelInit>, arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>> (arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&)>::operator()[abi:ue170006](arrow::compute::KernelContext*&&, arrow::compute::KernelInitArgs const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:193)
libarrow_python.dylib!std::__1::__function::__func<arrow::py::(anonymous namespace)::PythonUdfKernelInit, std::__1::allocator<arrow::py::(anonymous namespace)::PythonUdfKernelInit>, arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>> (arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&)>::operator()(arrow::compute::KernelContext*&&, arrow::compute::KernelInitArgs const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:364)
libarrow.1800.0.0.dylib!std::__1::__function::__value_func<arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>> (arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&)>::operator()[abi:ue170006](arrow::compute::KernelContext*&&, arrow::compute::KernelInitArgs const&) const (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:518)
libarrow.1800.0.0.dylib!std::__1::function<arrow::Result<std::__1::unique_ptr<arrow::compute::KernelState, std::__1::default_delete<arrow::compute::KernelState>>> (arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&)>::operator()(arrow::compute::KernelContext*, arrow::compute::KernelInitArgs const&) const (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:1169)
libarrow.1800.0.0.dylib!arrow::compute::(anonymous namespace)::BindNonRecursive(arrow::compute::Expression::Call, bool, arrow::compute::ExecContext*)::$_21::operator()() const (/Users/user/repos/python/arrow/cpp/src/arrow/compute/expression.cc:544)
libarrow.1800.0.0.dylib!arrow::compute::(anonymous namespace)::BindNonRecursive(arrow::compute::Expression::Call, bool, arrow::compute::ExecContext*) (/Users/user/repos/python/arrow/cpp/src/arrow/compute/expression.cc:560)
libarrow.1800.0.0.dylib!arrow::Result<arrow::compute::Expression> arrow::compute::(anonymous namespace)::BindImpl<arrow::Schema>(arrow::compute::Expression, arrow::Schema const&, arrow::compute::ExecContext*) (/Users/user/repos/python/arrow/cpp/src/arrow/compute/expression.cc:628)
libarrow.1800.0.0.dylib!arrow::Result<arrow::compute::Expression> arrow::compute::(anonymous namespace)::BindImpl<arrow::Schema>(arrow::compute::Expression, arrow::Schema const&, arrow::compute::ExecContext*) (/Users/user/repos/python/arrow/cpp/src/arrow/compute/expression.cc:608)
libarrow.1800.0.0.dylib!arrow::compute::Expression::Bind(arrow::Schema const&, arrow::compute::ExecContext*) const (/Users/user/repos/python/arrow/cpp/src/arrow/compute/expression.cc:641)
libarrow_substrait.1800.0.0.dylib!arrow::engine::FromProto(substrait::Rel const&, arrow::engine::ExtensionSet const&, arrow::engine::ConversionOptions const&) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/relation_internal.cc:629)
libarrow_substrait.1800.0.0.dylib!arrow::engine::DeserializePlan(arrow::Buffer const&, arrow::engine::ExtensionIdRegistry const*, arrow::engine::ExtensionSet*, arrow::engine::ConversionOptions const&) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/serde.cc:223)
libarrow_substrait.1800.0.0.dylib!arrow::engine::ExecuteSerializedPlan(arrow::Buffer const&, arrow::engine::ExtensionIdRegistry const*, arrow::compute::FunctionRegistry*, arrow::engine::ConversionOptions const&, bool, arrow::MemoryPool*) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/util.cc:48)

However, this is all happening in a with nogil Cython branch, which means that the thread state is NULL. The last GIL acquire/release cycle before the crash is happening due to this PyAcquireGIL.

Stripped stack trace to setting the thread state to `NULL`

libarrow_python.dylib!arrow::py::PyAcquireGIL::release() (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/common.h:124)
_substrait.cpython-313td-darwin.so!arrow::py::PyAcquireGIL::~PyAcquireGIL() (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/common.h:112)
_substrait.cpython-313td-darwin.so!arrow::py::PyAcquireGIL::~PyAcquireGIL() (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/common.h:112)
_substrait.cpython-313td-darwin.so!arrow::py::BoundFunction<arrow::acero::Declaration (_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>::Invoke(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) const (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/common.h:326)
_substrait.cpython-313td-darwin.so!std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)::operator()(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) const (/Users/user/repos/python/arrow/python/pyarrow/src/arrow/python/common.h:343)
_substrait.cpython-313td-darwin.so!decltype(std::declval<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>()(std::declval<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&>(), std::declval<arrow::Schema const&>())) std::__1::__invoke[abi:ue170006]<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::Result<arrow::acero::Declaration> (&&)(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__type_traits/invoke.h:340)
_substrait.cpython-313td-darwin.so!arrow::Result<arrow::acero::Declaration> std::__1::__invoke_void_return_wrapper<arrow::Result<arrow::acero::Declaration>, false>::__call[abi:ue170006]<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__type_traits/invoke.h:407)
_substrait.cpython-313td-darwin.so!std::__1::__function::__alloc_func<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), std::__1::allocator<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>, arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>::operator()[abi:ue170006](std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:193)
_substrait.cpython-313td-darwin.so!std::__1::__function::__func<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), std::__1::allocator<std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)> arrow::py::BindFunction<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), arrow::acero::Declaration, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&>(arrow::acero::Declaration (*)(_object*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&), _object*)::'lambda'(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>, arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>::operator()(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:364)
libarrow_substrait.1800.0.0.dylib!std::__1::__function::__value_func<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>::operator()[abi:ue170006](std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) const (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:518)
libarrow_substrait.1800.0.0.dylib!std::__1::function<arrow::Result<arrow::acero::Declaration> (std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&)>::operator()(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, arrow::Schema const&) const (/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/include/c++/v1/__functional/function.h:1169)
libarrow_substrait.1800.0.0.dylib!arrow::engine::FromProto(substrait::Rel const&, arrow::engine::ExtensionSet const&, arrow::engine::ConversionOptions const&) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/relation_internal.cc:423)
libarrow_substrait.1800.0.0.dylib!arrow::engine::FromProto(substrait::Rel const&, arrow::engine::ExtensionSet const&, arrow::engine::ConversionOptions const&) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/relation_internal.cc:611)
libarrow_substrait.1800.0.0.dylib!arrow::engine::DeserializePlan(arrow::Buffer const&, arrow::engine::ExtensionIdRegistry const*, arrow::engine::ExtensionSet*, arrow::engine::ConversionOptions const&) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/serde.cc:223)
libarrow_substrait.1800.0.0.dylib!arrow::engine::ExecuteSerializedPlan(arrow::Buffer const&, arrow::engine::ExtensionIdRegistry const*, arrow::compute::FunctionRegistry*, arrow::engine::ConversionOptions const&, bool, arrow::MemoryPool*) (/Users/user/repos/python/arrow/cpp/src/arrow/engine/substrait/util.cc:48)

Component(s)

Python

[jorisvandenbossche]

@lysnikolaou thanks for the report and analysis!

I am not super familiar with this part of our code base, but from reading your analysis, is it that we should ensure the GIL is held when calling that Py_INCREF ? (the usage of OwnedRefNoGIL only ensures to call PyAcquireGIL when the reference is decreased when going out of scope)

cc @pitrou

[lysnikolaou]

is it that we should ensure the GIL is held when calling that Py_INCREF ?

That's correct, yes!

[pitrou]

I'm taking a look at this.

[pitrou]

Issue resolved by pull request 43557
#43557

[lysnikolaou]

Thanks @pitrou!

[pitrou]

Thanks for your report @lysnikolaou , this helped us improve PyArrow CI and quality.