Bug: The logger swallows exceptions (such as KeyboardInterrupt)

[Qqwy]

PyO3's logger swallows exceptions.
For example, if while logging you press Ctrl+C, you get a traceback like for instance:

^CTraceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1651, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 366, in __init__
    self.process = os.getpid()
                   ^^^^^^^^^^^
KeyboardInterrupt

but then the program will continue running.

i.e. the output is akin to

...
DEBUG: Incrementing 250192 by 1...
DEBUG: Incrementing 250193 by 1...
DEBUG: Incrementing 250194 by 1...
DEBUG: Incrementing 250195 by 1...
DEBUG: Incrementing 250196 by 1...
^CTraceback (most recent call last):
  File "/usr/lib/python3.12/logging/__init__.py", line 1651, in makeRecord
    rv = _logRecordFactory(name, level, fn, lno, msg, args, exc_info, func,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/logging/__init__.py", line 366, in __init__
    self.process = os.getpid()
                   ^^^^^^^^^^^
KeyboardInterrupt
DEBUG: Incrementing 250198 by 1...
DEBUG: Incrementing 250199 by 1...
DEBUG: Incrementing 250200 by 1...
DEBUG: Incrementing 250201 by 1...
...

(The exact traceback message changes as it quits in the middle of different Python function calls, but the behaviour of the exception being swallowed remains the same.)

I try to have a well-behaved library that has some long-running calls in Rust and I want the user to be able to interrupt them using Ctrl+C. But py.check_signals() does not pick up an exception that was thrown during a call to the PyO3 logger.

Minimal Reproducible example repo here.

I previously asked about this behaviour in the PyO3 Discord group, and it was mentioned there that it most likely is a bug.

[vorner]

Do you have any information about what the library should be doing differently? I don't think I'm doing anything special about exceptions (so maybe some kind of dealing with them is missing) and the link to discord requires an account…

[Qqwy]

and the link to discord requires an account…

No worries! The Discord topic contains the same info as above, but with less detail (without the example).

Do you have any information about what the library should be doing differently? I don't think I'm doing anything special about exceptions

I believe that when the call to the Python logger returns an exception, it is turned into an Err(PyError) here.
However, this error is then printed and afterwards ignored here.

That particular Rust function cannot return the exception as an Err(...) because the log crate's Log trait requires logging to succeed with a ().
As such, the exception state needs to be restored as active exception outside of PyO3's Result handling. Probably using PyErr::restore or something similar to it.

[Qqwy]

I have created a PR that implements a fix.

Note that the minimal example also needs an alteration to pick this up:

Instead of

            py.check_signals()?;

we need:

            py.check_signals()?;
            match PyErr::take(py) {
                None => Ok(()),
                Some(err) => Err(err),
            }?;

because (contrary to what I thought) check_signals only turns signals into exceptions, it does not check for a pre-existing exception.