3.12.0b1 includes backwards incompatible change to operation of super()

[freakboy3742]

Bug report

The optimisation to LOAD_SUPER_ATTR introduced by #104270 appears to have introduced a backwards incompatibility.

The following code will reproduce the problem:

class MyInstance:
    def __new__(cls, ptr, name, bases, attrs):
        self = super().__new__(cls, name, bases, attrs)
        print(f"{MyInstance=} {self=} {type(self)=}, {super(MyInstance, type(self))=}")
        super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
        return self

    def __setattr__(self, name, value):
        raise Exception()


class MyClass(MyInstance, type):
    def __new__(cls, name):
        self = super().__new__(cls, id(name), name, (MyInstance,), {})

        return self


class1 = MyClass("Class1")
print(f"{class1.ptr=}")

class2 = MyClass("Class2")
print(f"{class2.ptr=}")

Under 3.12.0a7 (and previous stable Python versions going back to at least 3.7), this will succeed, outputting:

MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4364457392
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class2.ptr=4365761904

Under 3.12.0b1, it raises an error:

MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4370144336
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
Traceback (most recent call last):
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 24, in <module>
    class2 = MyClass("Class2")
             ^^^^^^^^^^^^^^^^^
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 16, in __new__
    self = super().__new__(cls, id(name), name, (MyInstance,), {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 7, in __new__
    super().__setattr__(self, "ptr", ptr)
TypeError:  expected 2 arguments, got 3

That is - the first MyClass instance can be instantiated; however, the second instance fails in mid-construction when invoking __setattr__. The state of the objects prior to the __setattr__ invocation appear to be identical, but the __setattr__ method behaves differently on the second invocation.

Git bisect has narrowed the cause down to #104270 (CC @carljm, @markshannon).

The reproduction case also fails if the call to super:

        super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)

is replaced with the simpler:

        super().__setattr__(self, "ptr", ptr)

Background

This test case has been extracted from rubicon-objc, causing beeware/rubicon-objc#313. Rubicon is a wrapper around the Objective C runtime used by macOS and iOS; MyInstance is an analog of ObjCInstance, and MyClass is a an analog of ObjCClass. ObjCInstance has an implementation of __setattr__ to redirect attribute access to the underlying ObjC calls. However, during construction, ObjCInstance needs to store a ptr of the underlying ObjC instance. This isn't a valid ObjC attribute, so super() is used to access the underlying __setattr__ implementation to set the attribute.

Your environment

• CPython versions tested on: 3.7.9, 3.10.11, 3.12.0a7, 3.12.0b1
• Operating system and architecture: macOS ARM64

Linked PRs

• gh-105035: fix super() calls on unusual types (e.g. meta-types) #105094
• [3.12] gh-105035: fix super() calls on unusual types (e.g. meta-types) (GH-105094) #105117

[sunmy2019]

Reduced test case

class MyInstance(type):
    pass


def test(name):
    self = MyInstance(name, (MyInstance,), {})
    super(MyInstance, type(self)).__setattr__(self, name, 1)


test("call1")
test("call2")  # <-- only at the second call

[sunmy2019]

Also, note the following passes. So it's definitely related to the interpreter core.

class MyInstance(type):
    pass


def test(name):
    self = MyInstance(name, (MyInstance,), {})

    m = super(MyInstance, type(self)).__setattr__
    m(self, name, id(name))


test("call1")
test("call2")

[sunmy2019]

The C call frame when the TypeError is raised

Details

#0  PyErr_Format (exception=0x555555bc11c0 <_PyExc_TypeError>, format=format@entry=0x555555a2b730 "%.200s expected %s%zd argument%s, got %zd") at Python/errors.c:1193
#1  0x000055555589f2ae in _PyArg_CheckPositional (name=name@entry=0x5555559cb7a6 "", nargs=nargs@entry=3, min=min@entry=2, max=max@entry=2) at Python/getargs.c:2813
#2  0x000055555589f344 in unpack_stack (args=0x7ffff7a23848, nargs=3, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2, vargs=vargs@entry=0x7fffffffd8b0) at Python/getargs.c:2836
#3  0x000055555589f478 in PyArg_UnpackTuple (args=<optimized out>, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2) at Python/getargs.c:2864
#4  0x0000555555796796 in wrap_setattr (self=0x555555db65d0, args=<optimized out>, wrapped=0x55555579cbd6 <type_setattro>) at Objects/typeobject.c:7972
#5  0x0000555555713412 in wrapperdescr_raw_call (kwds=0x0, args=0x7ffff7a23830, self=0x555555db65d0, descr=0x7ffff7b4cb90) at Objects/descrobject.c:537
#6  wrapperdescr_call (descr=descr@entry=0x7ffff7b4cb90, args=0x7ffff7a23830, args@entry=0x7ffff7a78470, kwds=kwds@entry=0x0) at Objects/descrobject.c:574
#7  0x0000555555702422 in _PyObject_MakeTpCall (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargs=<optimized out>, keywords=keywords@entry=0x0)
    at Objects/call.c:240
#8  0x0000555555702856 in _PyObject_VectorcallTstate (tstate=0x555555d21cd0 <_PyRuntime+478416>, callable=0x7ffff7b4cb90, args=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#9  0x0000555555702882 in PyObject_Vectorcall (callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:325
#10 0x0000555555865909 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, frame=0x7ffff7ae5090, throwflag=throwflag@entry=0) at Python/bytecodes.c:2643
#11 0x000055555586d3aa in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555d21cd0 <_PyRuntime+478416>) at ./Include/internal/pycore_ceval.h:87
#12 _PyEval_Vector (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, func=func@entry=0x7ffff7a184d0, locals=locals@entry=0x7ffff7a21250, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1610
#13 0x000055555586d468 in PyEval_EvalCode (co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/ceval.c:567
#14 0x00005555558decdf in run_eval_code_obj (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/pythonrun.c:1695
#15 0x00005555558df43f in run_mod (mod=mod@entry=0x555555dbff20, filename=filename@entry=0x7ffff7a22560, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, flags=flags@entry=0x7fffffffdea8, arena=arena@entry=0x7ffff7a63ee0)
    at Python/pythonrun.c:1716
#16 0x00005555558df54f in pyrun_file (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, start=start@entry=257, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, closeit=closeit@entry=1, 
    flags=0x7fffffffdea8) at Python/pythonrun.c:1616
#17 0x00005555558e2c7d in _PyRun_SimpleFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:433
#18 0x00005555558e2ed8 in _PyRun_AnyFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:78
#19 0x000055555590c13a in pymain_run_file_obj (program_name=program_name@entry=0x7ffff7a63d60, filename=filename@entry=0x7ffff7a22560, skip_source_first_line=0) at Modules/main.c:360
#20 0x000055555590c4d9 in pymain_run_file (config=config@entry=0x555555d04008 <_PyRuntime+356360>) at Modules/main.c:379
#21 0x000055555590d656 in pymain_run_python (exitcode=exitcode@entry=0x7fffffffe024) at Modules/main.c:610
#22 0x000055555590d6c6 in Py_RunMain () at Modules/main.c:689
#23 0x000055555590d73d in pymain_main (args=args@entry=0x7fffffffe080) at Modules/main.c:719
#24 0x000055555590d800 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#25 0x00005555556537e6 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

From the C side, it seems that
the first is called with ("call1", id(...))
the second is called with (self, "call2", id(...))

[sunmy2019]

The comment of _PyEval_EvalFrameDefault says

        // On entry, the stack is either
        //   [NULL, callable, arg1, arg2, ...]
        // or
        //   [method, self, arg1, arg2, ...]

And confirmed that in the latter case, the stack is

        //   [method, self, arg1, arg2, ...]

But it should be NULL?

We can dis this function, and we get

  5           0 RESUME                   0

  6           2 LOAD_GLOBAL              1 (NULL + MyInstance)
             12 LOAD_FAST                0 (name)
             14 LOAD_GLOBAL              0 (MyInstance)
             24 BUILD_TUPLE              1
             26 BUILD_MAP                0
             28 CALL                     3
             36 STORE_FAST               1 (self)

 12          38 LOAD_GLOBAL              2 (super)
             48 LOAD_GLOBAL              0 (MyInstance)
             58 LOAD_GLOBAL              5 (NULL + type)
             68 LOAD_FAST                1 (self)
             70 CALL                     1
             78 LOAD_SUPER_ATTR         15 (NULL|self + __setattr__)
             82 LOAD_FAST                1 (self)
             84 LOAD_FAST                0 (name)
             86 LOAD_CONST               1 (1)
             88 CALL                     3
             96 POP_TOP
             98 RETURN_CONST             0 (None)

             78 LOAD_SUPER_ATTR         15 (NULL|self + __setattr__)

must have something to do with this.

[carljm]

Thanks @freakboy3742, and @sunmy2019 for the simplified repro! I've identified the issue and working on a PR to fix.

Just for the record (and only because it confused me a bit until I double-checked), my testing shows that this wasn't broken by #104270, it was broken by the original introduction of LOAD_SUPER_ATTR in #103497.

[carljm]

This should be fixed now, in both 3.12 and main. Thanks for the report!

[freakboy3742]

Confirming this is now working for me. Thanks for the fast turnaround, @carljm!