Type slots are not extensible

[encukou]

Type slot functions (tp_new, nb_add) have fixed signatures, which makes it impossible to pass extra context. In particular

• you can't easily get an equivalent of Python's __class__ cell, so you can't easily emulate super in heap types.
• we can't pass a HPy-style context.
• we can't change the signature, e.g. when tp_getattro was added, tp_getattr was kept around, complicating inheritance.
• somewhat related, we can't easily implement flags like “this is a legacy function so it needs a GIL held” for nogil.
• (edit) docstrings / type annotations can't be set for type slot wrappers

[gvanrossum]

Isn't this a fundamental constraint of the C language? How would you go about fixing any of this?

[encukou]

Well, it's a problem, so it goes in this repo :)

The current implementation hits constraint of the C language. With a better API, it wouldn't have to. For example:

Leave the slots (and the whole type struct) out of the public API. Now we need to define getters and setters instead of direct struct access (which is a general theme around these parts).
For functions, most use cases don't really need getters: we need callers. Like PyNumber_Add for nb_add. For a new signature, we add a new caller -- and the old one gets/manufactures some default.
We don't have setters yet, but we can add them. (Well, we have PyType_Slot for initialization, but we can do much better than continuing to go that way.) The old setters would need to do a lot (relatively) of extra work -- setting a trampoline in the actual slot, and stashing the pointer to the user's function in some dynamically allocated space -- but that's OK for a deprecated function (especially when it's relatively easily to for users to update their code whenever they get annoyed by slowdowns/deprecation warnings).

This leaves out one use case: people use stuff like if (Py_TYPE(self)->tp_as_number.nb_add == my_add) for optimizations. I think it's solvable, but I haven't researched that use case enough yet.

[gvanrossum]

Ah, got it. You don't want to get rid of slots altogether, you want to get them out of the public API. I agree with that. Also the observation that you don't need getters but callers is useful.

[davidhewitt]

Also the observation that you don't need getters but callers is useful.

We already have PyType_GetSlot, which has served me well in the past. If we defined new slot constants which had extra context, I think we could still continue to use this given it returns void *. This can solve the cases like if (PyType_GetSlot(Py_TYPE(self), Py_nb_add) == my_add)

I think setters are less interesting because I don't think we want to mutate type objects after they've been initialised. PyType_Slot could be replaced with a similar struct which supports extra context.

A final bullet for the OP could be:

• Documentation / type annotations can't be set for type slot functions

[encukou]

If slots with new signatures have new names, we could still have (PyType_GetSlot(tp, Py_nb_add) == my_add) while the actual slot would be (PyType_GetSlot(tp, Py_nb_add_all_new_2024_edition) == trampoline_generated_by_python).

Mutating objects after initialization is necessary: you can do it in Python, so at least type.__setattr__ needs it.

[mattip]

Mutating objects after initialization is necessary

... as long as PyType_Modified is properly used ...

[encukou]

If we're replacing direct access to the type struct with setter functions, we get to ensure that.

[scoder]

Agreed that this is a problem. Passing (class) context into slot functions would be helpful.

[encukou]

Proposed “revolution” solution: capi-workgroup/api-revolution#4

(As I hinted above, I have a different approach to flesh out and propose, but will focus on “evolution” for the time being.)