incompatible behaviour between 3.12.0rc1 and previous versions in ctypes

[mattip]

Bug report

Testing out 3.12.0rc1 on NumPy, we came across numpy/numpy#24399. I have distilled this down to a reproducer that fails on 3.12.0rc1, but passes on previous versions.

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker, and am confident this bug has not been reported before

A clear and concise description of the bug

from ctypes import Structure, pointer, c_int16, cast, POINTER

class Struct(Structure):
    _fields_ = [('a', c_int16)]

Struct3 = 3 * Struct

c_array = (2 * Struct3)(
    Struct3(Struct(a=1), Struct(a=2), Struct(a=3)),
    Struct3(Struct(a=4), Struct(a=5), Struct(a=6))
)

assert c_array[0][0].a == 1

p_obj = cast(pointer(c_array), POINTER(pointer(c_array)._type_))
obj = p_obj.contents
# Works on 3.12, obj is 2d
assert obj[0][0].a == 1

p_obj = cast(pointer(c_array[0]), POINTER(pointer(c_array)._type_))
obj = p_obj.contents
# Fails on 3.12.0rc1, but works on 3.11 and earlier: obj is not 2d
assert obj[0][0].a == 1

Your environment

• CPython versions tested on: 3.12rc1
• Operating system and architecture: ubuntu 22.04

Edit: 3.12rc1 -> 3.12.0rc1

Linked PRs

• gh-107940: Only reuse the pointer object when it matches the _type_ of the container #108222

[mattip]

I see #107131 touched Pointer_get_contents to solve #46376. Perhaps that PR did not take cast into account when checking the equivalency of pointers?

[vstinner]

In the 3.12 branch, I confirm that the commit 54aaaad introduced the regression: issue #46376 of PR #107487 (backport of PR #107131).

[vstinner]

It seems like the PCP project is affected by this regression, see the Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=2231751

[vstinner]

I proposed a revert until the regression is understood and fixed: #108688

[ambv]

Revert for main merged. Revert for 3.12 up: #108864

[ambv]

3.12 revert landed as well.

[ambv]

3.11 was reverted before 3.11.5 in GH-108412.

[ambv]

We'll close this and continue search for a good solution in the original issue.

[vstinner]

We'll close this and continue search for a good solution in the original issue.

Thanks. IMO it's a reasonable practical solution to this complicated problem.

If someone comes up with a new fix, please ping me, I will try to review it! But I'm not available to investigate the issue and attempt writing a better ctypes fix.