pdb/bdb changes for PEP 669

[gaogaotiantian]

Feature or enhancement

Adapt pdb/bdb to PEP669

Pitch

After #103082, we will have the chance to build a much faster debugger. For breakpoints, we do not need to trigger trace function all the time and checking for the line number. Other optimizations for commands like return are available too.

The bad news is - it's almost impossible to do a completely backward compatible transition because the mechanism is quite different.

There are a couple of ways forward:

1. Create a new basic debugger based on the new mechanism. Leave bdb as it is and redesign pdb to use the new base debugger. Encourage users to migrate to the new base debugger while keep supporting bdb.

• pro: no historical burden, clean usage
• con: new module, potential duplicate code

2. Work on top of the current bdb, provide two ways to trigger the debugger. User can choose whether to use the old set_trace or the new mechanism.

• pro: incremental change, no new modules, reuse of the existing code
• con: the code could be messy, there will be two sets of APIs living in the same class with potential conflict

3. Just change bdb starting 3.12 to use the new mechanism and let users worry about their own tools built upon it.

• pro: code reuse still exists, life is much easier, enjoy the speed up immediately
• con: may not be an option

4. Leave everything there to use the old way.

• pro: no work need to be done at all
• con: no benefit from PEP 669 at all

5. Some other great idea that I did not think of

Previous discussion

https://peps.python.org/pep-0669/

Linked PRs

• gh-103103: Prototype of a new debugger based on PEP 669 #103496

[terryjreedy]

This should target 3.13. 3.12 beta1 is due May 8, less that 6 weeks and AFAIK 669 is not yet implemented. It will likely need some time before we can code anything using it. Besides which, touching bdb and pdb are not part of 669 and choosing one of the approaches above might need a PEP (or more than one if competing ideas).

I believe 3. is not an option. 4. has two versions:
4a. Leave bdb and pdb alone and leave anything new to 3rd parties.
4b. Leave bdb and pdb alone and write new 'bdb2' and 'pdb2' based on 669. Your 1 seems like leave bdb, write bdb2, replace bdb with bdb2.

About 5 years ago Cheryl Sabella and I added docstrings to bdb and likely edited it doc -- so we could better understand IDLE's bdb subclass, used for IDLE's visual debugger ('vdb'). Though I could be wrong, my memory is that bdb's code was not the best I have seen and might stand a rewrite. If a bdb re-write implemented the same API, then pdb would not need much change to use either old or new. But the API may have been part of what I did not like, and I have no idea at the moment whether it can sensibly remain as is (to make best use of the new 669 mechanism).

One of my back burner ideas has been to make vdb usable even if one were not using IDLE. I good time to explore this would be when revising vdb to use bdb2, however it ends up. To me, not having learned to use pdb, pdb looks old-fashioned and hard to use. I am pretty sure that many others in the same position react the same way.

[ambv]

This should target 3.13.

Yes but also no. We don't have to land the pdb/bdb changes for 3.12, I agree with that. But we need to start implementing them ASAP. It will be a huge missed opportunity if we don't exercise the PEP 669 implementation (which itself is currently being worked on) by implementing a debugger on top of that. That way we have a chance to discover issues early affecting a use case like a new pdb. Like, AsyncIO informed async/await and if it weren't for the built-in framework, I think it's unlikely we'd grow out of yield from.

Back to the question at hand, it seems to me that:

• option 3 is not feasible because there's too much potential for disruption;
• option 1 is similarly risky because there is tooling automating around Pdb as is;
• option 4 is, as I said, a missed opportunity at best;
• option 2 can be attempted but I expect it to be tricky to achieve without breaking any existing users of Bdb.

That leaves us essentially with Terry's "4b", which is to create a new fast debugger based on PEP 669. I think it's the right way forward as there are a few good opportunities here that would be hard to achieve with pdb. The biggest opportunity of PEP 669 isn't even the speed, it's the fact that a debugger built on top of it will automatically support all threads 🤯 That requires at least an extension of the command vocabulary of pdb but likely also requires changes to its output and potentially to the behavior of its commands.

Since that's a new project under a new module name, it's not a given it would end up in the standard library. And maybe it doesn't have to. We might go with a separate project within the Python organization on GitHub for this, like what MyPy is doing. This would allow us to use PyPI libraries in the implementation for stuff like auto-completion, syntax highlighting, text UIs, and so on.

If it works out well, at some point we might bring it into the standard library or bundle it like we do with pip. Definitely not a concern for now. What matters now is to start on the new debugger to test how PEP 669 feels. And a prerequisite of that is getting GH-103083 landed 🤞🏻

[gaogaotiantian]

pdb is a user interfacing layer so I don't see why we can't keep backward compatibility while switching the current bdb out. We can keep all the commands and functions(those are pretty standard for a debugger) so the existing tools relying on it won't be broken (or can be easily adapted).

A fancy debugger is great but I believe we have to have a command line working debugger in the standard libraries(battery included right?). The more 3rd-party libs we need (colorma? TUI-related stuff? auto-complete?), the harder it is to make into the standard libraries. There are already fancy debuggers out there like debugpy + IDE(VSCode, PyCharm), and an essential basic debugger should always be present.

That's what I was going to discuss in this issue originally. If we want to start a whole new project utilizing the new hooks to build a new debugger, that's totally fine, but what about pdb? It's a thing that users are already familiar with.

I guess the question here is - what would be different from the user's view between pdb2 and pdb? Are we going to change the debugger commands? Which commands are we not happy with? Are we going to change its interface? pdb.run() would be replaced? If everything will be kept the same, why do we need a pdb2 instead of changing pdb?

About multi-threading debugging, that's actually one of my next proposals for pdb. I don't think we "automatically" get the feature with PEP 669 and I think we have almost the same capability with the current infrastructure. Of course if we decided to abandon pdb then we only need to worry about PEP 669. There are some issues related, for example, stop-mode vs run-mode(whether all the threads are halted when we are debugging). How can we halt an arbitrary thread. I believe it's doable but it also lacks some essential work there. Therefore I'd suggest to discuss it in a separate issue.

[artemmukhin]

About multi-threading debugging, that's actually one of my next proposals for pdb. I don't think we "automatically" get the feature with PEP 669 and I think we have almost the same capability with the current infrastructure

@gaogaotiantian could you please provide more details about your concerns regarding PEP-669 and multithreading?

I have been experimenting with the implementation a little bit, and it appears that multithreading can be achieved out of the box. I have prepared a small demo based on the original example from @markshannon. Feel free to share your opinion and any further ideas.

Furthermore, I believe it would be great to provide good demos of how (and why) PEP-669 can be used in the monitoring tools before it is finally accepted and merged.

[gaogaotiantian]

@ortem because debugger is more than "triggering a function and do something". Yes with PEP 669 you can trigger a function in all threads, but you can do that in current design as well (with threading.settrace). The problem is - the debugger needs an interface.

Imagine you are debugging a thread, and a breakpoint of another thread hits - what will happen? Their behavior can not be symmetric because you only have one input channel. Or imaging one of the thread is constantly outputing strings, how could you debug other threads? Your debugger interface would be messed up by the outputs.

A real multithreading debugger needs to be able to switch to an arbitrary thread context and execute code/monitor variables in that context, how would a debugger do that?

To achieve this, the easiest way is to do "stop-mode" debugging, which means once we hit a breakpoint or a "stoppoint", halt all threads. In this case, we don't need to worry about impacts from all the other threads and we can maybe switch context without hassle. However, I don't believe Python has provided such a mechanism at the moment.

The more intuitive and also essential mode is "run-mode", where all the threads are running freely except for the one we debugged. We need a lot of extra care in this mode because of the reasons I mentioned above. This is quite a project. Preferably, we can halt arbitrary thread in the debugger to debug it.

All the stuff (which is just the tip of the iceburg) requires extra support from the under layer of Python and a lot of work. I don't believe it can be done "out of the box".

[markshannon]

Indeed, a thread-aware debugger is not a simple thing, but it should be possible with PEP 669.
It would need the callback functions for breakpoints and exceptions to be written in C in order to pause the thread and release the GIL, and the UI would need to run in its own thread.

@gaogaotiantian Do you think it would need extra support from the VM, or do you just mean that it cannot be done in pure Python?

[markshannon]

As for the original suggestion of porting pdb to PEP 669:

The bdb module's interface depends on sys.settrace() which makes it difficult to reimplement bdb on top of PEP 669.
It would be nice if we could, as it would fix both pdb and the idle debugger at the same time.

Reimplementing pdb on top PEP 669 should be possible, as the interface is higher level (although there is a set_trace() function).

Personally, I think it might be too late to do either for 3.12, but don't let me stop you trying.
The implementation is solid now, and should be merged soon, so you can start playing with it.

[gaogaotiantian]

I did some rethink on this matter, and it might be possible to do multi-thread debugging on pure python level. For stop mode, we can add a mutex for the debugger so only one thread has control to it. Every time we hit a breakpoint or stopped somewhere, we set the global LINE/INSTRUCTION event and in the callback all the threads need to get the lock before continue. This should equivalently halt all threads. We need some well-designed logic to restart the program - so users don't need to do continue on every single thread that's waiting for the lock, but that seems doable (and of course no one can be sure before implementing it).

However, I don't see a way to easily let an arbitrary thread run - PEP 669 is interpreter level, so we can't trigger/disable events on a specific event. That might eliminate the possibility of run mode.

From my personal perspective, even having only stop-mode debugging support for multi-thread is a huge improvement on current debugger.

And I agree, the time frame is too narrow for 3.12, we should shoot for 3.13. We also need time to figure out how we should proceed this (new bdb? whole new debugger?). Do we still support current pdb or do we still work on new features on it? Are we going to bring up a pdbx (Windows naming convention :)) to gradually replace pdb? A lot of the questions still remain open, and hopefully we can decide that before 3.13b.

In the meantime, I'm trying to polish the current pdb and hopefully bring some new features to it before 3.12b. #103124 and #103049 for example.

[gaogaotiantian]

Worked on a prototype debugger based on PEP 669. I think we can make the pdb interface the same as before if we wanted to. The events states need extra caring. I only implemented next, step and continue at this point and it worked fine. The more interesting thing would be the breakpoint.

[gaogaotiantian]

Some preliminary numbers for the debugger based on PEP 669. I had a basic implementation on breakpoints and tested with it.

def f(x):
    # Set breakpoint here
    x *= 2
    return x + 1

def fib(x):
    if x < 2:
        return 1
    return fib(x - 1) + fib(x - 2)

def test():
    start_time = time.time()
    fib(24)
    cost = time.time() - start_time
    print(cost)
    f(0)

test()

We had a time consuming function fib which has no breakpoint in it, and a function f that has a breakpoint but is not in the benchmark.

The test() function takes about 6ms with no debugger attached, and with PEP 669 debugger (with the breakpoint in f) the consumed time is basically identical.

With current pdb, it takes ~580ms because it needs to continuously check (for every line) whether the breakpoint is hit.

With VSCode debugger(debugpy), it takes ~65ms with or without the breakpoint. The debugger itself slows down the execution.

Overall the results looks promising.

[artemmukhin]

@fabioz the numbers above might be interesting to you. I expected debugpy's performance to be closer to the baseline, given that PEP 523 is being heavily leveraged.

[gaogaotiantian]

I had a very basic prototype for PEP 669 debugger now. I think we need to discuss if this is the right approach before I proceed more.

An extra layer(currently named MonitorGenie) is introduced to handle all monitoring related stuff. It knows about the breakpoints the debugger sets and the next action(step, next, etc.) and handle the monitoring events. With this layer, we can do almost everything at local event level - which is a huge performance improvement compared to the settrace solution.

However, for the breakpoint set on files, we still need to fire the PY_START event once for each code object until we can bind the breakpoint to a specific code object. This might be optimizable, but that's where it is now.

The current implementation still have two debugger classes - Bdbx and Pdbx. However, the responsibility changed from Bdb and Pdb. I moved more functionalities into Bdbx. Pdbx is closer to an interface now. All the data handling is in Bdbx.

One interesting thing to notice is that - we have tool id now. Not like we can do sys.settrace(None) and clean everything up anymore. For now Bdbx is a singleton(and a bit hacky one). We can change that if we have a better solution.

The interface between classes are clearer. MonitorGenie does a single callback to Bdbx to hand over the control. Bdbx also calls a single dispatch function to Pdbx. There are synchronized APIs between the classes (for example, set breakpoints).

As of now, the following pdb commands are implemented:

• continue
• step
• next
• return
• break (without condition)
• clear (only clear all)
• where
• quit

which I believe already comprise the most essential part of pdb - and more importantly, the part where PEP 669 makes a difference.

The code is still far from ready to be officially reviewed, but it's kind of working. I'd love some feedbacks from people (and preferably at least some core devs?) for the matter.

The most important ones are:

• Is this the right approach? Do we still want to continue on this?
• Do we want a new debugger at all? I mean yeah it would be nice to have one but that takes effort and time from core devs. It's also a bandwidth matter.
• I don't think 3.12 is feasible, but do we plan to do it in 3.13?

Performance wise, PEP 669 based debugger is crushing all the debuggers in the market, but it's not like we are trying to sell a product and have to hit the market fast :)

[markshannon]

@gaogaotiantian Thanks for working on this.

However, for the breakpoint set on files, we still need to fire the PY_START event once for each code object until we can bind the breakpoint to a specific code object. This might be optimizable, but that's where it is now.

The cost of once per code object should be effectively zero, or do you mean once per call to the code object? In which case returning DISABLE will get the cost down to once per code object.

One interesting thing to notice is that - we have tool id now.

I'd be inclined to unconditionally use DEBUGGER_ID, raising an exception if it is in use by another tool.

Performance wise, PEP 669 based debugger is crushing all the debuggers in the market

I'm very happy to hear that.

To answer your questions:

Is this the right approach? Do we still want to continue on this?

I think you are now the expert on this, so it is up to you 🙂

Do we want a new debugger at all? I mean yeah it would be nice to have one but that takes effort and time from core devs. It's also a bandwidth matter.

We will have to support bdb and pdb, so if we can fully replace them, then there is no additional maintenance, in fact there should be less as the new design is cleaner.

I don't think 3.12 is feasible, but do we plan to do it in 3.13?

Yes, it seems reasonable to target 3.13 for this.

[gaogaotiantian]

The cost of once per code object should be effectively zero, or do you mean once per call to the code object? In which case returning DISABLE will get the cost down to once per code object.

It's once per code object. The performance cost is non-observable at this point. I mentioned that because I don't know if there's any magic to just "get the code object".

I'd be inclined to unconditionally use DEBUGGER_ID, raising an exception if it is in use by another tool.

The issue I had here is - in the test, multiple debuggers are instantiated. Also it's possible in the future for the users to do multiple breakpoint() which could potentially instantiate multiple debuggers. Freeing the tool id when it's not needed seems like a fragile solution - the debugger has references to the frames so it's possible that gc is required to free the debugger. Singleton is a possible solution here to solve this issue. We can raise errors if the tool id is being used, but that would make the testing miserable.

Another possible solution is for the breakpoint() or it's pdbx equivalent break_here(), we check if a debugger is already created and use that debugger. It's like a singleton checking at specific entry. The users can create multiple debuggers if they can work with the tool ids.

We will have to support bdb and pdb, so if we can fully replace them, then there is no additional maintenance, in fact there should be less as the new design is cleaner.

Maintenance of pdb and bdb is fine (it's pretty stable now and does not require a lot of attentions for the recent couple of years). However, to make the new debugger releasable, there will be some effort from core-devs required. That's the time I was talking about.

[jdtsmith]

Having switched to Python for data-driven development from IDL, one thing I really miss about the latter is that the debugger is "always running". That is, to activate debugging in some code, just set a breakpoint. Stepping, continuing, etc. are just special functions.

Given how cheap PEP 669 makes few-breakpoint debugging, I wonder if this "always have a debugger running" mode is in the realm of possibility. Looking forward to the increased performance of (i)PDB.