Returning new instances of a derived type via a virtual factory function fails to keep the derived type alive

[yeswalrus]

Given a virtual function meant to return new instances of a type via a pointer to a base class, and a derived type declared in python, the return value cast won't keep the python object alive, resulting in a failure when attempting to call the virtual method on the new instance via the pointer. Because this function is called from c++, the return value policy has no effect.

Reproducible example code

https://github.com/yeswalrus/pybind11/tree/virtual-vector-return

[eacousineau]

I believe this may be the 5th issue that tracks this problem :P
Ultimately, I think it's a form of Object Slicing.

Related issues:

• Support transfer of ownership of Python subclasses of pybind11 C++ classes with shared_ptr<>, avoid slicing #1145 (mine, but using company acct)
• [BUG] Problem when creating derived Python objects from C++ (inheritance slicing) #1333
• Ownership of python objects inheriting from C++ classes #1389
• shared_ptrs to Python-derived instances outlive the associated Python object #1546

I've worked around this in our fork of pybind11, but have not yet figured out a good way to make it useful for mainstream consumption. (The py::wrapper class it introduces is... well... a lot of blech.)

While my patch above works and is conservative for object lifetime, it may be costly (in object code and possibly runtime performance). Also, the above code is strongly coupled to also supporting unique_ptr, which also induces more overhead.

Simpler workarounds, suggested in the aforementioned issues, point towards making something like keep_alive_forever, which could be done by stashing py::object inside the Python trampoline class. In this fashion, Python may never delete until the interpreter shuts down, but it could be "deletable" by C++.

As far as issue tracking goes, I'll try to see which thread has the best recommendations, and see if we can consolidate the current duplicates.

---

EDIT (2020/06/17):
Also:

• Factory class doesn't allow derived classes to be created #1120

[yeswalrus]

Ahh, sorry about the duplicate. Yeah, I dug into it in a debugger over the weekend a little bit - I was thinking something along the lines of making the trampoline class store a handle to the underlying python object would be a necessity.

[mmodenesi]

Reproducible example code

https://github.com/yeswalrus/pybind11/tree/virtual-vector-return

@yeswalrus what is the reproducible example, exactly? Thanks!

[EricCousineau-TRI]

Closing this in lieu of #1333 for tracking.