Skip to content

Commit

Permalink
Import current pybind/pybind11#5296: Enable type-safe interoperabilit…
Browse files Browse the repository at this point in the history
…y between different independent Python/C++ bindings systems. (#30158)

* `self.__cpp_transporter__()` proof of concept: Enable passing C++ pointers across extensions even if the `PYBIND11_INTERNALS_VERSION`s do not match.

* Include cleanup (mainly to resolve PyPy build failures).

* Fix clang-tidy errors.

* Resolve `error: extra

* factor out platform_abi_id.h from internals.h (no functional changes)

* factor out internals_version.h from internals.h (no functional changes)

* Update CMakeLists.txt, tests/extra_python_package/test_files.py

* Revert "factor out internals_version.h from internals.h (no functional changes)"

This reverts commit 3ccea8c.

* Remove internals_version.h from CMakeLists.txt, tests/extra_python_package/test_files.py

* `.__cpp_transporter__()` implementation: compare `pybind11_platform_abi_id`, `cpp_typeid_name`

* Add PremiumTraveler

* Rename test_cpp_transporter_traveler_type.h -> test_cpp_transporter_traveler_types.h

* Expand tests: `PremiumTraveler`, `get_points()`

* Shuffle order of tests (no real changes).

* Move `__cpp_transporter__` lambda to `py::cpp_transporter()` regular function.

* Use `type_caster_generic::load(self)` instead of `cast<T *>(self)`

* Pass `const std::type_info *` via `py::capsule` (instead of `cpp_typeid_name`).

* Make platform_abi_id.h completely stand-alone.

* rename exo_planet.cpp -> exo_planet_pybind11.cpp

* Add exo_planet_c_api.cpp (incomplete).

* Fix silly oversight (wrong filename in `#include`).

* Resolve clang-tidy errors:

```
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:10:18: error: 'wrapGetLuggage' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   10 | static PyObject *wrapGetLuggage(PyObject *, PyObject *) { return PyUnicode_FromString("TODO"); }
      | ~~~~~~           ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:14:20: error: 'ThisMethodDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   14 | static PyMethodDef ThisMethodDef[]
      | ~~~~~~             ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:17:27: error: 'ThisModuleDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   17 | static struct PyModuleDef ThisModuleDef = {
      | ~~~~~~                    ^
```

* Implement exo_planet_c_api GetLuggage(), GetPoints()

* Move new code from test_cpp_transporter_traveler_bindings.h to pybind11/detail/type_caster_base.h, under the name `class_dunder_cpp_transporter()`

* Fix oversight.

* Unconditionally add `__cpp_transporter__` method to all `py::class_` objects, but do not include that magic method in docstring signatures.

* Back out pybind11/detail/platform_abi_id.h for now. Maximizing reusability can be handled separately, later.

* Small cleanup.

* Restore and add to `test_call_cpp_transporter_*()`

* Ensure pybind/pybind11#3788 does not bite again.

* `class_dunder_cpp_transporter()`: replace `obj.cast<std::string>()` with `std::string(obj)`

* Add (simple) copyright notices in all newly added files.

* Globally replace cpp_transporter with cpp_conduit

* style: pre-commit fixes

* IWYU fixes

* Rename `class_dunder_cpp_conduit()` -> `cpp_conduit_method()`

* Change `pybind11_platform_abi_id`, `pointer_kind` argument types from `str` to `bytes`.

This avoids the unicode decode/encode roundtrips:

* More robust (no decode/encode errors).

* Minor runtime optimization.

* Systematically rename `cap_cpp_type_info` -> `cpp_type_info_capsule` (no functional changes).

* Systematically replace `cpp_type_info_capsule` `name`: `"const std::type_info *"` -> `typeid(std::type_info).name()` (this IS a functional change).

This provides an extra layer of protection against C++ ABI mismatches:

* The first and most important layer is that the `PYBIND11_PLATFORM_ABI_ID`s must match between extensions.

* The second layer is that the `typeid(std::type_info).name()`s must match between extensions.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
Ralf W. Grosse-Kunstleve and pre-commit-ci[bot] authored Sep 1, 2024
1 parent 7d51202 commit 4841661
Show file tree
Hide file tree
Showing 14 changed files with 462 additions and 10 deletions.
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ endif()
set(PYBIND11_HEADERS
include/pybind11/detail/class.h
include/pybind11/detail/common.h
include/pybind11/detail/cpp_conduit.h
include/pybind11/detail/descr.h
include/pybind11/detail/dynamic_raw_ptr_cast_if_possible.h
include/pybind11/detail/function_record_pyobject.h
Expand Down
77 changes: 77 additions & 0 deletions include/pybind11/detail/cpp_conduit.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
// Copyright (c) 2024 The pybind Community.

#pragma once

#include <pybind11/pytypes.h>

#include "common.h"
#include "internals.h"

#include <typeinfo>

PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
PYBIND11_NAMESPACE_BEGIN(detail)

// Forward declaration needed here: Refactoring opportunity.
extern "C" inline PyObject *pybind11_object_new(PyTypeObject *type, PyObject *, PyObject *);

inline bool type_is_managed_by_our_internals(PyTypeObject *type_obj) {
#if defined(PYPY_VERSION)
auto &internals = get_internals();
return bool(internals.registered_types_py.find(type_obj)
!= internals.registered_types_py.end());
#else
return bool(type_obj->tp_new == pybind11_object_new);
#endif
}

inline bool is_instance_method_of_type(PyTypeObject *type_obj, PyObject *attr_name) {
PyObject *descr = _PyType_Lookup(type_obj, attr_name);
return bool((descr != nullptr) && PyInstanceMethod_Check(descr));
}

inline object try_get_cpp_conduit_method(PyObject *obj) {
if (PyType_Check(obj)) {
return object();
}
PyTypeObject *type_obj = Py_TYPE(obj);
str attr_name("__cpp_conduit__");
bool assumed_to_be_callable = false;
if (type_is_managed_by_our_internals(type_obj)) {
if (!is_instance_method_of_type(type_obj, attr_name.ptr())) {
return object();
}
assumed_to_be_callable = true;
}
PyObject *method = PyObject_GetAttr(obj, attr_name.ptr());
if (method == nullptr) {
PyErr_Clear();
return object();
}
if (!assumed_to_be_callable && PyCallable_Check(method) == 0) {
Py_DECREF(method);
return object();
}
return reinterpret_steal<object>(method);
}

inline void *try_raw_pointer_ephemeral_from_cpp_conduit(handle src,
const std::type_info *cpp_type_info) {
object method = try_get_cpp_conduit_method(src.ptr());
if (method) {
capsule cpp_type_info_capsule(const_cast<void *>(static_cast<const void *>(cpp_type_info)),
typeid(std::type_info).name());
object cpp_conduit = method(bytes(PYBIND11_PLATFORM_ABI_ID),
cpp_type_info_capsule,
bytes("raw_pointer_ephemeral"));
if (isinstance<capsule>(cpp_conduit)) {
return reinterpret_borrow<capsule>(cpp_conduit).get_pointer();
}
}
return nullptr;
}

#define PYBIND11_HAS_CPP_CONDUIT

PYBIND11_NAMESPACE_END(detail)
PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE)
10 changes: 6 additions & 4 deletions include/pybind11/detail/internals.h
Original file line number Diff line number Diff line change
Expand Up @@ -353,15 +353,17 @@ struct type_info {
# define PYBIND11_INTERNALS_KIND ""
#endif

#define PYBIND11_PLATFORM_ABI_ID \
PYBIND11_INTERNALS_KIND PYBIND11_COMPILER_TYPE PYBIND11_STDLIB PYBIND11_BUILD_ABI \
PYBIND11_BUILD_TYPE

#define PYBIND11_INTERNALS_ID \
"__pybind11_internals_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION) \
PYBIND11_INTERNALS_KIND PYBIND11_COMPILER_TYPE PYBIND11_STDLIB \
PYBIND11_BUILD_ABI PYBIND11_BUILD_TYPE "__"
PYBIND11_PLATFORM_ABI_ID "__"

#define PYBIND11_MODULE_LOCAL_ID \
"__pybind11_module_local_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION) \
PYBIND11_INTERNALS_KIND PYBIND11_COMPILER_TYPE PYBIND11_STDLIB \
PYBIND11_BUILD_ABI PYBIND11_BUILD_TYPE "__"
PYBIND11_PLATFORM_ABI_ID "__"

/// Each module locally stores a pointer to the `internals` data. The data
/// itself is shared among modules with the same `PYBIND11_INTERNALS_ID`.
Expand Down
10 changes: 7 additions & 3 deletions include/pybind11/detail/try_as_void_ptr_capsule_get_pointer.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
PYBIND11_NAMESPACE_BEGIN(detail)

// Forward declaration needed here: Refactoring opportunity.
extern "C" inline PyObject *pybind11_object_new(PyTypeObject *type, PyObject *, PyObject *);

// To avoid conflicts with PR pybind/pybind11#5296 before it is merged.
PYBIND11_NAMESPACE_BEGIN(pybind11clif)

// Replace all occurrences of substrings in a string.
inline void replace_all(std::string &str, const std::string &from, const std::string &to) {
if (str.empty()) {
Expand All @@ -28,9 +34,6 @@ inline void replace_all(std::string &str, const std::string &from, const std::st
}
}

// Forward declaration needed here: Refactoring opportunity.
extern "C" inline PyObject *pybind11_object_new(PyTypeObject *type, PyObject *, PyObject *);

inline bool type_is_pybind11_class_(PyTypeObject *type_obj) {
#if defined(PYPY_VERSION)
auto &internals = get_internals();
Expand Down Expand Up @@ -86,5 +89,6 @@ inline void *try_as_void_ptr_capsule_get_pointer(handle src, const char *typeid_

#define PYBIND11_HAS_TRY_AS_VOID_PTR_CAPSULE_GET_POINTER

PYBIND11_NAMESPACE_END(pybind11clif)
PYBIND11_NAMESPACE_END(detail)
PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE)
43 changes: 42 additions & 1 deletion include/pybind11/detail/type_caster_base.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <pybind11/trampoline_self_life_support.h>

#include "common.h"
#include "cpp_conduit.h"
#include "descr.h"
#include "dynamic_raw_ptr_cast_if_possible.h"
#include "internals.h"
Expand All @@ -23,8 +24,10 @@
#include "value_and_holder.h"

#include <cstdint>
#include <cstring>
#include <iterator>
#include <new>
#include <stdexcept>
#include <string>
#include <type_traits>
#include <typeindex>
Expand Down Expand Up @@ -993,14 +996,22 @@ class type_caster_generic {
}
return false;
}
bool try_cpp_conduit(handle src) {
value = try_raw_pointer_ephemeral_from_cpp_conduit(src, cpptype);
if (value != nullptr) {
return true;
}
return false;
}

bool try_as_void_ptr_capsule(handle src) {
#ifdef PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT
// The `as_void_ptr_capsule` feature is needed for PyCLIF-SWIG interoperability
// in the Google-internal environment, but the current implementation is lacking
// any safety checks. To lower the risk potential, the feature is activated
// only if the smart_holder is used (PyCLIF-pybind11 uses `classh`).
if (typeinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
value = try_as_void_ptr_capsule_get_pointer(src, cpptype->name());
value = pybind11clif::try_as_void_ptr_capsule_get_pointer(src, cpptype->name());
if (value != nullptr) {
return true;
}
Expand Down Expand Up @@ -1139,6 +1150,10 @@ class type_caster_generic {
return true;
}

if (convert && cpptype && this_.try_cpp_conduit(src)) {
return true;
}

if (convert && cpptype && this_.try_as_void_ptr_capsule(src)) {
return true;
}
Expand Down Expand Up @@ -1170,6 +1185,32 @@ class type_caster_generic {
void *value = nullptr;
};

inline object cpp_conduit_method(handle self,
const bytes &pybind11_platform_abi_id,
const capsule &cpp_type_info_capsule,
const bytes &pointer_kind) {
#ifdef PYBIND11_HAS_STRING_VIEW
using cpp_str = std::string_view;
#else
using cpp_str = std::string;
#endif
if (cpp_str(pybind11_platform_abi_id) != PYBIND11_PLATFORM_ABI_ID) {
return none();
}
if (std::strcmp(cpp_type_info_capsule.name(), typeid(std::type_info).name()) != 0) {
return none();
}
if (cpp_str(pointer_kind) != "raw_pointer_ephemeral") {
throw std::runtime_error("Invalid pointer_kind: \"" + std::string(pointer_kind) + "\"");
}
const auto *cpp_type_info = cpp_type_info_capsule.get_pointer<const std::type_info>();
type_caster_generic caster(*cpp_type_info);
if (!caster.load(self, false)) {
return none();
}
return capsule(caster.value, cpp_type_info->name());
}

/**
* Determine suitable casting operator for pointer-or-lvalue-casting type casters. The type caster
* needs to provide `operator T*()` and `operator T&()` operators.
Expand Down
6 changes: 4 additions & 2 deletions include/pybind11/pybind11.h
Original file line number Diff line number Diff line change
Expand Up @@ -608,7 +608,8 @@ class cpp_function : public function {
int index = 0;
/* Create a nice pydoc rec including all signatures and
docstrings of the functions in the overload chain */
if (chain && options::show_function_signatures()) {
if (chain && options::show_function_signatures()
&& strcmp(rec->name, "__cpp_conduit__") != 0) {
// First a generic signature
signatures += rec->name;
signatures += "(*args, **kwargs)\n";
Expand All @@ -617,7 +618,7 @@ class cpp_function : public function {
// Then specific overload signatures
bool first_user_def = true;
for (auto *it = chain_start; it != nullptr; it = it->next) {
if (options::show_function_signatures()) {
if (options::show_function_signatures() && strcmp(rec->name, "__cpp_conduit__") != 0) {
if (index > 0) {
signatures += '\n';
}
Expand Down Expand Up @@ -1955,6 +1956,7 @@ class class_ : public detail::generic_type {
= instances[std::type_index(typeid(type))];
});
}
def("__cpp_conduit__", cpp_conduit_method);
}

template <typename Base, detail::enable_if_t<is_base<Base>::value, int> = 0>
Expand Down
2 changes: 2 additions & 0 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ set(PYBIND11_TEST_FILES
test_const_name
test_constants_and_functions
test_copy_move
test_cpp_conduit
test_custom_type_casters
test_custom_type_setup
test_descr_src_loc
Expand Down Expand Up @@ -245,6 +246,7 @@ tests_extra_targets("test_exceptions.py;test_local_bindings.py;test_stl.py;test_
# And add additional targets for other tests.
tests_extra_targets("test_exceptions.py" "cross_module_interleaved_error_already_set")
tests_extra_targets("test_gil_scoped.py" "cross_module_gil_utils")
tests_extra_targets("test_cpp_conduit.py" "exo_planet_pybind11;exo_planet_c_api")
tests_extra_targets("test_exc_namespace_visibility.py"
"namespace_visibility_1;namespace_visibility_2")

Expand Down
103 changes: 103 additions & 0 deletions tests/exo_planet_c_api.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
// Copyright (c) 2024 The pybind Community.

// THIS MUST STAY AT THE TOP!
#include <pybind11/pybind11.h> // EXCLUSIVELY for PYBIND11_PLATFORM_ABI_ID
// Potential future direction to maximize reusability:
// (e.g. for use from SWIG, Cython, PyCLIF, nanobind):
// #include <pybind11/compat/platform_abi_id.h>
// This would only depend on:
// 1. A C++ compiler, WITHOUT requiring -fexceptions.
// 2. Python.h

#include "test_cpp_conduit_traveler_types.h"

#include <Python.h>
#include <typeinfo>

namespace {

void *get_cpp_conduit_void_ptr(PyObject *py_obj, const std::type_info *cpp_type_info) {
PyObject *cpp_type_info_capsule
= PyCapsule_New(const_cast<void *>(static_cast<const void *>(cpp_type_info)),
typeid(std::type_info).name(),
nullptr);
if (cpp_type_info_capsule == nullptr) {
return nullptr;
}
PyObject *cpp_conduit = PyObject_CallMethod(py_obj,
"__cpp_conduit__",
"yOy",
PYBIND11_PLATFORM_ABI_ID,
cpp_type_info_capsule,
"raw_pointer_ephemeral");
Py_DECREF(cpp_type_info_capsule);
if (cpp_conduit == nullptr) {
return nullptr;
}
void *void_ptr = PyCapsule_GetPointer(cpp_conduit, cpp_type_info->name());
Py_DECREF(cpp_conduit);
if (PyErr_Occurred()) {
return nullptr;
}
return void_ptr;
}

template <typename T>
T *get_cpp_conduit_type_ptr(PyObject *py_obj) {
void *void_ptr = get_cpp_conduit_void_ptr(py_obj, &typeid(T));
if (void_ptr == nullptr) {
return nullptr;
}
return static_cast<T *>(void_ptr);
}

extern "C" PyObject *wrapGetLuggage(PyObject * /*self*/, PyObject *traveler) {
const auto *cpp_traveler
= get_cpp_conduit_type_ptr<pybind11_tests::test_cpp_conduit::Traveler>(traveler);
if (cpp_traveler == nullptr) {
return nullptr;
}
return PyUnicode_FromString(cpp_traveler->luggage.c_str());
}

extern "C" PyObject *wrapGetPoints(PyObject * /*self*/, PyObject *premium_traveler) {
const auto *cpp_premium_traveler
= get_cpp_conduit_type_ptr<pybind11_tests::test_cpp_conduit::PremiumTraveler>(
premium_traveler);
if (cpp_premium_traveler == nullptr) {
return nullptr;
}
return PyLong_FromLong(static_cast<long>(cpp_premium_traveler->points));
}

PyMethodDef ThisMethodDef[] = {{"GetLuggage", wrapGetLuggage, METH_O, nullptr},
{"GetPoints", wrapGetPoints, METH_O, nullptr},
{nullptr, nullptr, 0, nullptr}};

struct PyModuleDef ThisModuleDef = {
PyModuleDef_HEAD_INIT, // m_base
"exo_planet_c_api", // m_name
nullptr, // m_doc
-1, // m_size
ThisMethodDef, // m_methods
nullptr, // m_slots
nullptr, // m_traverse
nullptr, // m_clear
nullptr // m_free
};

} // namespace

#if defined(WIN32) || defined(_WIN32)
# define EXO_PLANET_C_API_EXPORT __declspec(dllexport)
#else
# define EXO_PLANET_C_API_EXPORT __attribute__((visibility("default")))
#endif

extern "C" EXO_PLANET_C_API_EXPORT PyObject *PyInit_exo_planet_c_api() {
PyObject *m = PyModule_Create(&ThisModuleDef);
if (m == nullptr) {
return nullptr;
}
return m;
}
10 changes: 10 additions & 0 deletions tests/exo_planet_pybind11.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Copyright (c) 2024 The pybind Community.

#if defined(PYBIND11_INTERNALS_VERSION)
# undef PYBIND11_INTERNALS_VERSION
#endif
#define PYBIND11_INTERNALS_VERSION 900000001

#include "test_cpp_conduit_traveler_bindings.h"

PYBIND11_MODULE(exo_planet_pybind11, m) { pybind11_tests::test_cpp_conduit::wrap_traveler(m); }
1 change: 1 addition & 0 deletions tests/extra_python_package/test_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
detail_headers = {
"include/pybind11/detail/class.h",
"include/pybind11/detail/common.h",
"include/pybind11/detail/cpp_conduit.h",
"include/pybind11/detail/descr.h",
"include/pybind11/detail/dynamic_raw_ptr_cast_if_possible.h",
"include/pybind11/detail/function_record_pyobject.h",
Expand Down
Loading

0 comments on commit 4841661

Please sign in to comment.