asyncio.run sometimes hangs with cancelled subprocesses #125502
Comments
|
Interestingly, removing the |
|
the tasks seems to be getting stuck in cpython/Lib/asyncio/unix_events.py Line 215 in 92af191 |
|
the problem occurs when asyncio.runners._cancel_all_tasks is run at an inopportune instant when connecting pipes: This task gets cancelled: cpython/Lib/asyncio/base_subprocess.py Line 56 in 92af191 which means self._pending_calls is never run: cpython/Lib/asyncio/base_subprocess.py Lines 199 to 202 in 92af191 so when _try_finish appends self._call_connection_lost to self._pending_calls: cpython/Lib/asyncio/base_subprocess.py Line 257 in 92af191 call_connection_lost is never called, which means self._exit_waiters are never woken: cpython/Lib/asyncio/base_subprocess.py Lines 259 to 270 in 92af191 Here's a demo that hangs every time for me: import sys
import inspect
import asyncio
from subprocess import PIPE
async def run_sleep():
proc = await asyncio.create_subprocess_exec(
"sleep",
"0.002",
stdout=PIPE,
)
await proc.communicate()
async def amain():
loop = asyncio.get_running_loop()
task = asyncio.current_task(loop)
coro = task.get_coro()
called_cancel = False
def cancel_eventually():
my_coro = coro
while inspect.iscoroutine(my_coro.cr_await):
my_coro = my_coro.cr_await
if my_coro.cr_code is loop._make_subprocess_transport.__code__:
print("_cancel_all_tasks")
tasks = asyncio.all_tasks()
for task in tasks:
task.cancel()
else:
loop.call_soon(cancel_eventually)
loop.call_soon(cancel_eventually)
await run_sleep()
def main():
asyncio.run(amain())
if __name__ == "__main__":
sys.exit(main()) |
|
@obeattie you can work around this problem by using a TaskGroup: import asyncio
import logging
from subprocess import PIPE
async def run_sleep():
proc = await asyncio.create_subprocess_exec(
"sleep",
"0.002",
stdout=PIPE,
)
await proc.communicate()
async def run_loop():
print("-- run_loop")
async def run_sleep_forever():
while True:
await run_sleep()
try:
async with asyncio.timeout(1), asyncio.TaskGroup() as tg:
for _ in range(20):
tg.create_task(run_sleep_forever())
except TimeoutError:
pass
finally:
print(f"-- exiting run_loop")
if __name__ == "__main__":
logging.basicConfig(level=logging.DEBUG)
asyncio.run(run_loop()) |
|
Thanks for confirming this is an issue and for the workaround @graingert. I already wired up a workaround which is a bit different to this – though I am sure others who hit this will appreciate that until the issue is fixed. |
|
Duplicate of #103847 |
Bug report
Bug description:
When using asyncio to launch subprocesses, and cancelling them before they are done,
asyncio.runsometimes hangs forever.I can reproduce this fairly reliably on both macOS and Linux, though you may need to run this script a few times to hit the hang, so run it in a loop (eg.
for i in $(seq 1 10); do python repro.py; done):When you hit this issue, you will see the program hangs forever instead of exiting. If I send
SIGTERM, I can see a traceback like this, showing it's stuck waiting for coroutines to finish cancelling:CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
The text was updated successfully, but these errors were encountered: