Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-35810: Incref heap-allocated types in PyObject_Init #11661

Merged
merged 10 commits into from
Mar 27, 2019
107 changes: 60 additions & 47 deletions Doc/whatsnew/3.8.rst
Original file line number Diff line number Diff line change
Expand Up @@ -469,8 +469,10 @@ Build and C API Changes
``1`` for objects implementing ``__index__()``.
(Contributed by Serhiy Storchaka in :issue:`36048`.)

* Heap-allocated Types will now incref through :c:func:`PyObject_Init` (and
its parallel macro PyObject_INIT) instead of :c:func:`PyType_GenericAlloc`.
* Heap-allocated type objects will now increase their reference count
in :c:func:`PyObject_Init` (and its parallel macro ``PyObject_INIT``)
instead of in :c:func:`PyType_GenericAlloc`. Types that modify instance
allocation or deallocation may need to be adjusted.
(Contributed by Eddie Elizondo in :issue:`35810`.)


Expand Down Expand Up @@ -668,51 +670,62 @@ Changes in the Python API
Changes in the C API
--------------------------

The incref to a heap-allocated type has been moved from
:c:func:`PyType_GenericAlloc` to :c:func:`PyObject_Init` and
:c:func:`PyObject_INIT`. This now makes types created through
:c:func:`PyType_FromSpec` behave like any other class in managed code.

For the vast majority of cases, there should be no side effect. However, this
causes types that are manually incref after allocating an instance to become
immortal. Furthermore, the incref guarantee should now be reflected by having
a decref during instance deallocation.

To correctly port these types into 3.8, please apply the following changes:

* Remove any manual incref on the itself after allocating an instance - if any.
This happens after calling :c:func:`PyObject_New`, :c:func:`PyObject_NewVar`,
:c:func:`PyObject_GC_New`, :c:func:`PyObject_GC_NewVar`, or any other custom
allocator that uses :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`.

Example::

static foo_struct *
foo_new(PyObject *type) {
foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type);
if (foo == NULL)
return NULL;
#if PY_VERSION_HEX < 0x03080000
PY_INCREF(type)
#endif
return foo;
}

* Add a decref to the type in any custom tp_dealloc. The type should always
decref at deallocation time.

Example::

static void
foo_dealloc(foo_struct *instance) {
PyObject *type = Py_TYPE(instance);
PyObject_GC_Del(instance);
#if PY_VERSION_HEX >= 0x03080000
Py_DECREF(type);
#endif
}

(Contributed by Eddie Elizondo in :issue:`35810`.)
* Instances of heap-allocated types (such as those created with
:c:func:`PyType_FromSpec`) hold a reference to their type object.
Increasing the reference count of these type objects has been moved from
:c:func:`PyType_GenericAlloc` to the more low-level functions,
:c:func:`PyObject_Init` and :c:func:`PyObject_INIT`.
This makes types created through :c:func:`PyType_FromSpec` behave like
other classes in managed code.

Statically allocated types are not affected.

For the vast majority of cases, there should be no side effect.
However, types that manually increase the reference count after allocating
an instance (perhaps to work around the bug) may now become immortal.
To avoid this, these classes need to call Py_DECREF on the type object
during instance deallocation.

To correctly port these types into 3.8, please apply the following
changes:

* Remove :c:macro:`Py_INCREF` on the type object after allocating an
instance - if any.
This may happen after calling :c:func:`PyObject_New`,
:c:func:`PyObject_NewVar`, :c:func:`PyObject_GC_New`,
:c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses
:c:func:`PyObject_Init` or :c:func:`PyObject_INIT`.

Example::

static foo_struct *
foo_new(PyObject *type) {
foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type);
if (foo == NULL)
return NULL;
#if PY_VERSION_HEX < 0x03080000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Cython, we normally use the exact alpha/beta version where the change was introduced (to support testing during the CPython release cycle), but we obviously don't know that yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example is not really written for projects with Cython's level of involvement. Most projects will only test against a nightly build or the latest alpha/beta (if even that). The few others should know to adjust the example. We don't explicitly tell you to rename foo_new, either :)

// Workaround for Python issue 35810; no longer necessary in Python 3.8
PY_INCREF(type)
#endif
return foo;
}

* Ensure that all custom ``tp_dealloc`` functions of heap-allocated types
decrease the type's reference count.

Example::

static void
foo_dealloc(foo_struct *instance) {
PyObject *type = Py_TYPE(instance);
PyObject_GC_Del(instance);
#if PY_VERSION_HEX >= 0x03080000
// This was not needed before Python 3.8 (Python issue 35810)
Py_DECREF(type);
#endif
}

(Contributed by Eddie Elizondo in :issue:`35810`.)


CPython bytecode changes
Expand Down