Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v2.11] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. #5370

Merged
merged 2 commits into from
Sep 13, 2024
Merged

[v2.11] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. #5370

merged 2 commits into from
Sep 13, 2024

Conversation

rwgk
Copy link
Collaborator

@rwgk rwgk commented Sep 13, 2024

Description

Backport of PR #5296 to the pybind11 v2.11 series, to maximize the benefit of the new cross-version interoperability feature.

Suggested changelog entry:

Backport of PR #5296: The pybind11 v2.11 series now interoperates with other pybind11 versions that include PR #5296. (See the description of PR #5296 for limitations.)

rwgk and others added 2 commits September 13, 2024 11:01
@rwgk @pre-commit-ci
@henryiii @Skylion007
Enable type-safe interoperability between different independent Pytho…
f52b8f8 
…n/C++ bindings systems. (pybind#5296)

* `self.__cpp_transporter__()` proof of concept: Enable passing C++ pointers across extensions even if the `PYBIND11_INTERNALS_VERSION`s do not match.

* Include cleanup (mainly to resolve PyPy build failures).

* Fix clang-tidy errors.

* Resolve `error: extra

* factor out platform_abi_id.h from internals.h (no functional changes)

* factor out internals_version.h from internals.h (no functional changes)

* Update CMakeLists.txt, tests/extra_python_package/test_files.py

* Revert "factor out internals_version.h from internals.h (no functional changes)"

This reverts commit 3ccea8c.

* Remove internals_version.h from CMakeLists.txt, tests/extra_python_package/test_files.py

* `.__cpp_transporter__()` implementation: compare `pybind11_platform_abi_id`, `cpp_typeid_name`

* Add PremiumTraveler

* Rename test_cpp_transporter_traveler_type.h -> test_cpp_transporter_traveler_types.h

* Expand tests: `PremiumTraveler`, `get_points()`

* Shuffle order of tests (no real changes).

* Move `__cpp_transporter__` lambda to `py::cpp_transporter()` regular function.

* Use `type_caster_generic::load(self)` instead of `cast<T *>(self)`

* Pass `const std::type_info *` via `py::capsule` (instead of `cpp_typeid_name`).

* Make platform_abi_id.h completely stand-alone.

* rename exo_planet.cpp -> exo_planet_pybind11.cpp

* Add exo_planet_c_api.cpp (incomplete).

* Fix silly oversight (wrong filename in `#include`).

* Resolve clang-tidy errors:

```
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:10:18: error: 'wrapGetLuggage' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   10 | static PyObject *wrapGetLuggage(PyObject *, PyObject *) { return PyUnicode_FromString("TODO"); }
      | ~~~~~~           ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:14:20: error: 'ThisMethodDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   14 | static PyMethodDef ThisMethodDef[]
      | ~~~~~~             ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:17:27: error: 'ThisModuleDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   17 | static struct PyModuleDef ThisModuleDef = {
      | ~~~~~~                    ^
```

* Implement exo_planet_c_api GetLuggage(), GetPoints()

* Move new code from test_cpp_transporter_traveler_bindings.h to pybind11/detail/type_caster_base.h, under the name `class_dunder_cpp_transporter()`

* Fix oversight.

* Unconditionally add `__cpp_transporter__` method to all `py::class_` objects, but do not include that magic method in docstring signatures.

* Back out pybind11/detail/platform_abi_id.h for now. Maximizing reusability can be handled separately, later.

* Small cleanup.

* Restore and add to `test_call_cpp_transporter_*()`

* Ensure pybind#3788 does not bite again.

* `class_dunder_cpp_transporter()`: replace `obj.cast<std::string>()` with `std::string(obj)`

* Add (simple) copyright notices in all newly added files.

* Globally replace cpp_transporter with cpp_conduit

* style: pre-commit fixes

* IWYU fixes

* Rename `class_dunder_cpp_conduit()` -> `cpp_conduit_method()`

* Change `pybind11_platform_abi_id`, `pointer_kind` argument types from `str` to `bytes`.

This avoids the unicode decode/encode roundtrips:

* More robust (no decode/encode errors).

* Minor runtime optimization.

* Systematically rename `cap_cpp_type_info` -> `cpp_type_info_capsule` (no functional changes).

* Systematically replace `cpp_type_info_capsule` `name`: `"const std::type_info *"` -> `typeid(std::type_info).name()` (this IS a functional change).

This provides an extra layer of protection against C++ ABI mismatches:

* The first and most important layer is that the `PYBIND11_PLATFORM_ABI_ID`s must match between extensions.

* The second layer is that the `typeid(std::type_info).name()`s must match between extensions.

* Fix sort order accident in tests/CMakeLists.txt

* Apply suggestions from code review

Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

* style: pre-commit fixes

* refactor: rename to _pybind_conduit_v1_

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* Add test_home_planet_wrap_very_lonely_traveler(), test_exo_planet_pybind11_wrap_very_lonely_traveler()

* Resolve clang-tidy errors:

```
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:39:32: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
   10 |     py::class_<LonelyTraveler>(m, "LonelyTraveler");
      |                                ^
      |                                std::move( )
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:43:52: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
   43 |     py::class_<VeryLonelyTraveler, LonelyTraveler>(m, "VeryLonelyTraveler");
      |                                                    ^
      |                                                    std::move( )
```

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
@rwgk
Remove from __future__ import annotations
5df1db5
@rwgk rwgk mentioned this pull request Sep 13, 2024
Closed
@rwgk
Copy link
Collaborator Author

rwgk commented Sep 13, 2024

Squash-merging, after coordinating with @henryiii on slack.

@rwgk rwgk marked this pull request as ready for review September 13, 2024 21:52
@rwgk rwgk requested a review from henryiii as a code owner September 13, 2024 21:52
@rwgk rwgk merged commit 854cf72 into pybind:v2.11 Sep 13, 2024
68 of 83 checks passed
@rwgk rwgk deleted the pybind11_conduit_v1_backport_v2_11 branch September 13, 2024 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@henryiii henryiii Awaiting requested review from henryiii henryiii is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rwgk