Staying in the interpreter and avoiding calls to C code that call back into Python

[markshannon]

Whatever means of execution we choose, and whatever techniques we use to optimize, switching contexts is to be avoided.
Switching contexts is expensive in terms of data movement and potentially even more expensive in terms of lost optimization potential.

For example, https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagram/ describes the lengths that the cinder team go to when inlining functions to avoid the cost of context switching at call boundaries.

Regardless of whether we have a JIT compiler, or what form it may take, we want to avoid the context switch from the interpreter to C code where possible.

These context switches can be be (loosely) categorized into:

1. Implementation of language components in C that call back into the interpreter, e.g. gen.send(). gen.__next__(), etc.
2. Builtin functions that call back into the interpreter next(), any(), all(), enumerate(), etc.
3. Third-party extensions that perform dynamic operations (usually in the form C-API calls)

We can remove 1. by adding new bytecodes, or new specialization of existing bytecodes.
We can fix 2. by judiciously re-implementing builtin functions in Python.
It is case 3 that will be problematic. It will take a while, which is why we should start sooner, rather than later, even though the benefits might not show up until later versions of Python (probably 3.13). Cython is the obvious place to start, but we should look to other important projects as well.

Not all calls to C code are bad. If the amount of work done is significant, and there is little to no dispatching, that is fine.
E.g. deque.append(x) treats its argument as opaque. Converting it to Python would gain nothing, even with an effective JIT.

More complex cases are modules like pyexpat, which has a lot of callbacks into Python code, are going to be tougher to deal with.
To avoid the undesirable context switching would need the parsing code to be wrapped in a generator, to allow top-level Python code to call into the parser and not pass in callbacks. This would not be a trivial task.