Mute 3.12.2 error

[jlowin]

Python 3.12.2 introduced an issue that our background log handlers sometimes hit when writing logs while the system is tearing down (which happens very frequently in CI and interactive runs). The issue appears to be fixed in Python 3.12.3. If the problem happens, there is no recourse and it is classified as a Python bug, not a Prefect bug, so muting this specific error will hopefully save devs from the red herring rabbit hole I spent the last day in.

[bunchesofdonald]

Thank you!

[zzstoatzz]

dang, i also went down this rabbit hole for a bit -- thank you!

[zzstoatzz]

just to surface some sync discussion

i still think there's a lurking problem with this part of QueueService, since even in python 3.12.3, we get the following

--- Logging error ---
Traceback (most recent call last):
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/logging/handlers.py", line 145, in emit
    APILogWorker.instance().send(log)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/logging/handlers.py", line 79, in instance
    return super().instance(*settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/services.py", line 248, in instance
    cls._instances[key] = cls._new_instance(*args)
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/services.py", line 268, in _new_instance
    from_sync.call_soon_in_loop_thread(create_call(instance.start)).result()
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/calls.py", line 312, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/calls.py", line 182, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/calls.py", line 346, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nate/github.com/prefecthq/prefect/src/prefect/_internal/concurrency/services.py", line 71, in start
    _register_atexit(self._at_exit)
  File "/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1559, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

[jlowin]

@zzstoatzz good call. I'm not sure if this issue intersects that or is just extremely related to that. Nonetheless as an experiment I modified the log worker to flush in the global thread instead of a new thread; I'm curious to see the result. However even the traceback you supplied already shows the error happening when call_soon_in_loop_thread which means that it might not be related to creating new threads as much as it is interacting with them (e.g. registering atexit)

[jlowin]

I think given that tests are passing lets merge this to close one avenue and continue exploring what @zzstoatzz surfaced separately