Propagating cancellation through threads

[njsmith]

While writing up #607, I was thinking about a neat feature that curio has. I'm not sure if it's very important, but I envy it a bit, so I might as well open an issue to dump thoughts out where we can see them.

The feature is: say you use run_sync_in_worker_thread to enter a thread, and then from the thread you use BlockingTrioPortal.run to re-enter trio. So conceptually, this is all one stack, that starts in trio, goes into a thread, and then goes back into trio. Now suppose the original run_sync_in_worker_thread is cancelled. It would be neat if this caused the code inside the BlockingTrioPortal.run call to raise a Cancelled error that propagated all the way back out of trio, through the thread, and back into trio.

See #607 for more details about why this is tricky, and hard to fix even by extending the current cancel API.

I guess either we need some way to reenter a given cancel binding (which feels very weird and unlikely to have other use cases), or else some way to capture the exception in run_sync_in_worker_thread (<- this part we have) and then inject it into the eventual task (<- this part we don't have). Well... unless we do it by making a new cancel scope inside the eventual task, cancel it, and then re-raise after the inner cancellation unwinds. (We could even fudge the traceback if we want to be really tricky.) Or... we could switch from attaching Cancelled objects to specific scopes/bindings, and make it so that each with block checks if it is associated with the outermost cancelled scope, and if so it catches all Cancelled exceptions, and then it would be fine to manually capture a Cancelled from inside the re-entry task and pass it over to the thread, before it gets caught. Except, ugh, that won't work if the exception isn't Cancelled but rather KeyboardInterrupt.

It wouldn't be terrible to manually cancel the re-entry task, then call the raise_cancel function to get the real exception, and attach the internal exception to it as a __context__ (though this would require #285).

Alternatively... can we avoid all this rigmorale? What if run_sync_in_worker_thread created a nursery, and when BTP.run re-entered it put tasks into that nursery, instead of the system nursery? Then any cancel scope that contained the run_sync_in_worker_thread call would also automatically contain any re-entry tasks.

KeyboardInterrupt might still require some careful handling, since it doesn't follow the normal cancel scope scoping rules. I guess we could do some shenanigans like pushing the actual wait-for-thread into a child task? Actually this would be pretty simple: open nursery, start a child to do the actual thread waiting stuff, then park in the nursery __aexit__ – KeyboardInterrupt will be delivered to __aexit__ and cause the nursery to be cancelled. (Actually, this is an interesting possible strategy for delivering KeyboardInterrupt in general: pick a nursery and inject it directly there, as if it were raised by a task inside that nursery. Or do this to the system nursery, though that feels weird.) I'd be a little nervous about adding overhead to run_sync_in_worker_thread, but maybe it wouldn't be that bad.

[njsmith]

In retrospect, I think I was making this too complicated. The case where we want to propagate cancellation is when we're inside a trio.to_thread.run_sync worker thread. And in that case, we already have a task on the Trio side: the one that called to_thread.run_sync. It already has the right cancellation context. And it's just sitting around doing nothing, waiting for the thread. So we might as well give it something to do.

Basically, after spawning the thread, it would sit in a loop like:

while True:
    msg_from_thread = await wait_task_rescheduled(...)
    if msg_from_thread.type is THREAD_DONE:
        return msg_from_thread.result.unwrap()
    elif msg_from_thread.type is RUN:
        result = await outcome.acapture(msg_from_thread.async_fn)
        to_thread_queue.put(result)
    ...

We'd stash the communications channels in thread-local variables, like we do the trio token, and then the thread would send messages back to report when it's finished, or when it has some work it needs to do in trio context.

Interesting special cases that we'll want to make sure we handle correctly:

• Entering Trio from a foreign thread (in this case we do need to spawn a new task, like now)
• cancellable=True + attempting to re-enter into a task that's gone