feat: enable type-safe interoperability between different independent Python/C++ bindings systems.

[rwgk]

Description

This PR enables type-safe interoperability between different independent Python/C++ bindings systems, including pybind11 versions with different PYBIND11_INTERNALS_VERSIONs. That means, innovations that require pybind11 internals version bumps become far less disruptive, almost to a negligible degree for most practical purposes (see NOTE below for the limitation).

More concretely, this PR enables extracting raw C++ pointers from Python instances of py::class_-wrapped types.

The implementation involves a

def _pybind11_conduit_v1_(
    self,
    pybind11_platform_abi_id: bytes,
    cpp_type_info_capsule: capsule,
    pointer_kind: bytes) -> capsule

method that is added unconditionally to all py::class_-wrapped types (to avoid "oh, too late!" situations). — The implementation of this method is identical for all types, i.e. it is not templated, therefore it compiles very quickly and adds very little binary size overhead. I.e. the cost is very small, while the benefit of maximizing interoperability is expected to be very large.

The design of the _pybind11_conduit_v1_ feature provides two layers of protection against C++ ABI mismatches:

• The first and most important layer is that the pybind11_platform_abi_ids must match between extensions. — This will never be perfect, but is the same pragmatic approach used in pybind11 since 2017 (PYBIND11_INTERNALS_ID).

• The second layer is that the typeid(std::type_info).name()s must match between extensions.

This PR includes calls of the _pybind11_conduit_v1_() method from:

• pybind11::detail::type_caster_generic (in pybind11/detail/type_caster_base.h, with helper functions in pybind11/detail/cpp_conduit.h)

• tests/exo_planet_c_api.cpp

The latter is a great way to quickly learn how the _pybind11_conduit_v1_ feature works. exo_planet_c_api.cpp only uses the plain CPython API, not any pybind11 features, except one macro resolving to a C++ string literal: PYBIND11_PLATFORM_ABI_ID.

The integration into type_caster_generic means that any object, from any Python/C++ bindings system (e.g. in theory at the moment SWIG, Cython, PyCLIF, nanobind) providing a compatible self._pybind11_conduit_v1_() method can be passed from Python to pybind11-wrapped C++ functions.

For entertainment: This PR makes pirate-like approaches to getting to the wrapped C++ pointers obsolete. Here is a particularly creative piece of code enabling SWIG-pybind11 interoperability, exactly as found in the wild, in an undisclosed location.

NOTE: Matching PYBIND11_INTERNALS_IDs are still critically important for this situation:

• py::class_<base> in one extension,
• py::class_<derived, base> in another.

For a demonstration, please see test_home_planet_wrap_very_lonely_traveler() and test_exo_planet_pybind11_wrap_very_lonely_traveler() in tests/test_cpp_conduit.py.

---

This PR passed Google-global testing:

• merged into the smart_holder branch: internal ID OCL:669841401:BASE:669907294:1725198946622:db3091a3
• merged into pybind11clif, then running with PyCLIF-pybind11: internal ID OCL:669954247:BASE:670062753:1725250171762:cb476a31

---

Suggestions for reviewing this PR (or just understanding what it does in detail):

• Start with test/exo_planet_c_api.cpp

• Review cpp_conduit_method() in pybind11/detail/type_caster_base.h, which is the only implementation in this PR building an object to be returned by self._pybind11_conduit_v1_(). — Other bindings systems would have their own implementations.

• The most interesting part of this PR is this small code fragment in cpp_conduit_method():

    const auto *cpp_type_info = cap_cpp_type_info.get_pointer<const std::type_info>();
    type_caster_generic caster(*cpp_type_info);
    if (!caster.load(self, false)) {
        return none();
    }

Note that this takes care of all inheritance-related pointer conversions (up- and down-casts).

• Review the rest of the changes in include/pybind11/detail/type_caster_base.h, which calls helper functions in:

• include/pybind11/detail/cpp_conduit.h

• Review the very simple change in include/pybind11/detail/internals.h, which just factors out PYBIND11_PLATFORM_ABI_ID from PYBIND11_INTERNALS_ID.

• Review the very simple changes in include/pybind11/pybind11.h. — Note that the auto-magically added _pybind11_conduit_v1_ method is not included in docstring signatures, to not clutter the docstrings.

• Review the added tests.

---

Beyond this PR:

---

Please take a quick look at the top of tests/exo_planet_c_api.cpp.

It is unfortunate that bindings systems other than pybind11 need to #include <pybind11/pybind11.h> just to get to the PYBIND11_PLATFORM_ABI_ID macro (which resolves to a simple C++ string literal). This leads to significant problems in environments that have C++ exception handling disabled (e.g. via -fno-exceptions).

An attempt was made to establish an alternative that maximizes reusability, but the implementation requires refactoring of pybind11/detail/common.h and pybind11/detail/internals.h. One particular difficulty is that PYBIND11_PLATFORM_ABI_ID can only be built after #include <Python.h>, because that transitively #includes system headers that define some of the system macros used in building PYBIND11_PLATFORM_ABI_ID. To not make this fairly simple PR significantly more complicated, the changes for the attempt were backed out. The pure refactoring work is best handled in a follow-on PR.

FWIW: In an ideal world, PYBIND11_PLATFORM_ABI_ID would be spelled as PY_CPP_COMMUNITY_PLATFORM_ABI_ID or similar, and come in a tiny additional header file included with CPython releases, so that all Python/C++ bindings systems could standardize on it.

---

Currently b"raw_pointer_ephemeral" (see above) is the only supported pointer_kind. Ideas for the future go in the direction of b"std::shared_ptr", b"std::unique_ptr", which will probably be relatively simple to add after the smart_holder branch is merged. — It is not clear (to rwgk@) if those additions will be worth the effort and added complexity, but having pointer_kind from the start is expressive and leaves the door open for future developments. — Additionally, the pointer_kind could be leveraged for implementing a b"__diagnose__" (or similar) feature, to report why retrieving a desired C++ pointer is failing.

---

mypy stubgenc will generate this additional signature:

    def _pybind11_conduit_v1_(self, *args, **kwargs) -> Any: ...

If this is undesired by the community (?), the signature could be suppressed similar to how dunder methods are suppressed here.

---

For completeness, the starting point for this PR was PR #3633 & PR #3788, although this PR is very different in many details.

Suggested changelog entry:

A new ``self._pybind11_conduit_v1_()`` method is automatically added to all ``py::class_``-wrapped types, to enable type-safe interoperability between different independent Python/C++ bindings systems, including pybind11 versions with different ``PYBIND11_INTERNALS_VERSION``s.

[rwgk]

Thanks a lot @henryiii for jumping in!

I just updated the PR description, to systematically replace __cpp_conduit__ with _pybind_conduit_v1_, and a minor edit related to mypy (since the new method name isn't a dunder name anymore it'll need something more special in mypy).

I'll now play a bit with py::class_<base> in one extension, py::class_<derived, base> in another, with non-matching PYBIND11_INTERNALS_ID, to validate my understanding, before maybe adding an explanation to the PR description. — I don't expect that this will lead to any production code changes.

[rwgk]

I'll now play a bit with py::class_<base> in one extension, py::class_<derived, base> in another, with non-matching PYBIND11_INTERNALS_ID, to validate my understanding, before maybe adding an explanation to the PR description. — I don't expect that this will lead to any production code changes.

Done: 22dbfa6

@henryiii if this looks good to you, please merge.

I volunteer to backport this to 2.11 and 2.12.

[rwgk]

Continuing the fact finding here, picking up from:

• @rwgk: [Inductor] Fix Triton tests after updating pybind11 to 2.13.6 pytorch/pytorch#136280 (comment)

• @henryiii: [Inductor] Fix Triton tests after updating pybind11 to 2.13.6 pytorch/pytorch#136280 (comment)

https://docs.python.org/3/reference/lexical_analysis.html#identifiers

Any use of * names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.

Emphasis is from the original docs.

I interpret that as: Use with caution. — That's in line with what Thomas Wouters told me (re-quoting): "They're not reserved, but you do run the risk of clashing with the runtime. There's other third-party packages that use their own dunders."

Concluding from the reference documentation that dunders are "reserved" is a big leap. I believe stating that in isolation is very easily misleading.

I'm not trying to convince anyone to adopt a specific interpretation, but I believe it's important to not jump to conclusions. @henryiii I hope you will edit your comments here and under the pytorch PR to clarify, to make it less likely that people take your interpretation as source of truth.

Beyond that, I'm ok waiting to see if there is any other fallout, although I'm still very much in favor of adopting the dunder convention as soon as possible. With the pybind11 part in the name there's really no danger of clashing with the runtime. Currently we're the odd ones out with the leading-and-trailing single underscore (AFAIK).

[henryiii]

PyTorch was filtering on __* to try to pick out only data members. This is incorrect; it just happens to work if the only methods are Python built-in methods, which all are __dunder__ methods. If someone added a private data member (__*), this would incorrectly filter it out. If someone added a dunder name that stored a member value (like __dict__), this is wrong, too. This needs to be fixed in PyTorch, and doesn't mean we were wrong in choosing this name.

The point I'm making is that __dunder__ can't be the standard way to define a protocol method, because Python states any (emphasis in original source) usage in any context is subject to breakage without warning. Yes, it doesn't use the word "reserved", but isn't that the definition of reserved? Python can break any usage that is not defined already by Python at any time? How else would you define reserved?

Other libraries absolutely do use _sunder_ names for protocols. The Jupyter display protocol is one example, it uses things like _repr_html_(). They usually are used for single-library or dedicated protocols. Python's enum module uses _sunder_ names too, and I'm not the first person to refer to __dunder__ as reserved.

Other libraries also do make their own __dunder__ names too, a popular example being NumPy. These are generally used to make cross-library protocols, like __array_function__ - I suspect looking like it's part of Python itself makes this look "official". While I think that's technically incorrect, and Python would be within their rights to break __array_function__ tomorrow, it's really unlikely because the fallout would be high.

Once we have a standard mechanism for this, we could propose using a __dunder__ name; I think if we make a SPEC, that might be enough. But while it's pybind11 specific, I think we should stick with non-dunder names, unless at least one more (and hopefully more valid) example of breakage comes up. The PyTorch issue would have just happened to work because we would have been filtered the same way as the poor filtering that was already present.

If we redo this, we need to handle reading the old values too, as we've already released and packages have already been built, so it's not as simple as just swapping the name out.