Raising an error inside a task of a task group can produce: `RuntimeError: called 'started' twice on the same task status` #706

dominik-schwabe · 2024-03-27T22:03:06Z

Things to check first

I have searched the existing issues and didn't find my bug already reported there
I have checked that my bug is still present in the latest release

AnyIO version

4.3.0

Python version

3.12.2

What happened?

Calling task_status.started() can raise RuntimeError: called 'started' twice on the same task status under certain circumstances (see examples), even though it was not called twice.
The reason for this is that the underlying future gets awaited in the start() method. When the start() method gets canceled, the underlying future gets also canceled which makes calling set_result on the future later fail and raise the erroneous RuntimeError.
This can be fixed by using asyncio.shield when awaiting the future.

anyio/src/anyio/_backends/_asyncio.py

Line 782 in 05a3d02

return await future

How can we reproduce the bug?

from anyio import create_task_group, run
from anyio.abc import TaskStatus


async def taskfunc(*, task_status: TaskStatus):
    task_status.started()
    raise ValueError


async def main():
    async with create_task_group() as tg:
        await tg.start(taskfunc)
        await tg.start(taskfunc)


run(main)

or

from anyio import create_task_group, run
from anyio.abc import TaskStatus


async def taskfunc(*, task_status: TaskStatus):
    task_status.started()


async def main():
    async with create_task_group() as tg:
        coro = tg.start(taskfunc)
        tg.cancel_scope.cancel()
        await coro


run(main)

The text was updated successfully, but these errors were encountered:

agronholm · 2024-03-28T10:07:25Z

asyncio.shield() is an anti-pattern I would very much like to stay away from. I'll get to the bottom of this over Easter.

dominik-schwabe · 2024-03-28T10:17:43Z

Understandable. Alternatively one could check if the future got cancled and only raise an error if this is not the case.

anyio/src/anyio/_backends/_asyncio.py

Line 621 in 05a3d02

self._future.set_result(value)

agronholm · 2024-03-28T14:34:38Z

Understandable. Alternatively one could check if the future got cancled and only raise an error if this is not the case.

anyio/src/anyio/_backends/_asyncio.py

Line 621 in 05a3d02

self._future.set_result(value)

Sounds much better already.

dominik-schwabe · 2024-03-28T15:12:48Z

I modified my pull request with the alternative solution, but I am not sure if the code on line 628 and 629 should also be executed when the future is cancelled.
https://github.com/dominik-schwabe/anyio/blob/d9179b4750b49ab593c6692e30c8c62f7e295dc4/src/anyio/_backends/_asyncio.py#L619-L629

agronholm · 2024-03-28T20:28:47Z

No worries, I'll have a look within a couple days.

jonathanslenders · 2024-03-28T20:42:21Z

Thanks! I just see this thread; having exactly the same issue. :-) Let me know if there's anything I can help with or test.

davidbrochart · 2024-04-03T14:42:49Z

I just got hit by this bug, and @dominik-schwabe's fix solved it.

Fixes #706.

Bumps [anyio](https://github.com/agronholm/anyio) from 4.3.0 to 4.4.0. <details> <summary>Release notes</summary> Sourced from <a href="https://github.com/agronholm/anyio/releases">anyio's releases</a>. <blockquote> <h2>4.4.0</h2> <ul> <li>Added the <code>BlockingPortalProvider</code> class to aid with constructing synchronous counterparts to asynchronous interfaces that would otherwise require multiple blocking portals</li> <li>Added <code>__slots__</code> to <code>AsyncResource</code> so that child classes can use <code>__slots__</code> (<a href="https://redirect.github.com/agronholm/anyio/pull/733">#733</a>; PR by Justin Su)</li> <li>Added the <code>TaskInfo.has_pending_cancellation()</code> method</li> <li>Fixed erroneous <code>RuntimeError: called 'started' twice on the same task status</code> when cancelling a task in a TaskGroup created with the <code>start()</code> method before the first checkpoint is reached after calling <code>task_status.started()</code> (<a href="https://redirect.github.com/agronholm/anyio/issues/706">#706</a>; PR by Dominik Schwabe)</li> <li>Fixed two bugs with <code>TaskGroup.start()</code> on asyncio: <ul> <li>Fixed erroneous <code>RuntimeError: called 'started' twice on the same task status</code> when cancelling a task in a TaskGroup created with the <code>start()</code> method before the first checkpoint is reached after calling <code>task_status.started()</code> (<a href="https://redirect.github.com/agronholm/anyio/issues/706">#706</a>; PR by Dominik Schwabe)</li> <li>Fixed the entire task group being cancelled if a <code>TaskGroup.start()</code> call gets cancelled (<a href="https://redirect.github.com/agronholm/anyio/issues/685">#685</a>, <a href="https://redirect.github.com/agronholm/anyio/issues/710">#710</a>)</li> </ul> </li> <li>Fixed a race condition that caused crashes when multiple event loops of the same backend were running in separate threads and simultaneously attempted to use AnyIO for their first time (<a href="https://redirect.github.com/agronholm/anyio/issues/425">#425</a>; PR by David Jiricek and Ganden Schaffner)</li> <li>Fixed cancellation delivery on asyncio incrementing the wrong cancel scope's cancellation counter when cascading a cancel operation to a child scope, thus failing to uncancel the host task (<a href="https://redirect.github.com/agronholm/anyio/issues/716">#716</a>)</li> <li>Fixed erroneous <code>TypedAttributeLookupError</code> if a typed attribute getter raises <code>KeyError</code></li> <li>Fixed the asyncio backend not respecting the <code>PYTHONASYNCIODEBUG</code> environment variable when setting the <code>debug</code> flag in <code>anyio.run()</code></li> <li>Fixed <code>SocketStream.receive()</code> not detecting EOF on asyncio if there is also data in the read buffer (<a href="https://redirect.github.com/agronholm/anyio/issues/701">#701</a>)</li> <li>Fixed <code>MemoryObjectStream</code> dropping an item if the item is delivered to a recipient that is waiting to receive an item but has a cancellation pending (<a href="https://redirect.github.com/agronholm/anyio/issues/728">#728</a>)</li> <li>Emit a <code>ResourceWarning</code> for <code>MemoryObjectReceiveStream</code> and <code>MemoryObjectSendStream</code> that were garbage collected without being closed (PR by Andrey Kazantcev)</li> <li>Fixed <code>MemoryObjectSendStream.send()</code> not raising <code>BrokenResourceError</code> when the last corresponding <code>MemoryObjectReceiveStream</code> is closed while waiting to send a falsey item (<a href="https://redirect.github.com/agronholm/anyio/issues/731">#731</a>; PR by Ganden Schaffner)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> Sourced from <a href="https://github.com/agronholm/anyio/blob/master/docs/versionhistory.rst">anyio's changelog</a>. <blockquote> <h1>Version history</h1> This library adheres to <code>Semantic Versioning 2.0 <http://semver.org/></code>_. 4.4.0 <ul> <li> Added the <code>BlockingPortalProvider</code> class to aid with constructing synchronous counterparts to asynchronous interfaces that would otherwise require multiple blocking portals </li> <li> Added <code>__slots__</code> to <code>AsyncResource</code> so that child classes can use <code>__slots__</code> (<code>[#733](agronholm/anyio#733) <https://github.com/agronholm/anyio/pull/733></code>_; PR by Justin Su) </li> <li> Added the <code>TaskInfo.has_pending_cancellation()</code> method </li> <li> Fixed erroneous <code>RuntimeError: called 'started' twice on the same task status</code> when cancelling a task in a TaskGroup created with the <code>start()</code> method before the first checkpoint is reached after calling <code>task_status.started()</code> (<code>[#706](agronholm/anyio#706) <https://github.com/agronholm/anyio/issues/706></code>_; PR by Dominik Schwabe) </li> <li> Fixed two bugs with <code>TaskGroup.start()</code> on asyncio: <ul> <li>Fixed erroneous <code>RuntimeError: called 'started' twice on the same task status</code> when cancelling a task in a TaskGroup created with the <code>start()</code> method before the first checkpoint is reached after calling <code>task_status.started()</code> (<code>[#706](agronholm/anyio#706) <https://github.com/agronholm/anyio/issues/706></code>_; PR by Dominik Schwabe)</li> <li>Fixed the entire task group being cancelled if a <code>TaskGroup.start()</code> call gets cancelled (<code>[#685](agronholm/anyio#685) <https://github.com/agronholm/anyio/issues/685></code>, <code>[#710](agronholm/anyio#710) <https://github.com/agronholm/anyio/issues/710></code>)</li> </ul> </li> <li> Fixed a race condition that caused crashes when multiple event loops of the same backend were running in separate threads and simultaneously attempted to use AnyIO for their first time (<code>[#425](agronholm/anyio#425) <https://github.com/agronholm/anyio/issues/425></code>_; PR by David Jiricek and Ganden Schaffner) </li> <li> Fixed cancellation delivery on asyncio incrementing the wrong cancel scope's cancellation counter when cascading a cancel operation to a child scope, thus failing to uncancel the host task (<code>[#716](agronholm/anyio#716) <https://github.com/agronholm/anyio/issues/716></code>_) </li> <li> Fixed erroneous <code>TypedAttributeLookupError</code> if a typed attribute getter raises <code>KeyError</code> </li> <li> Fixed the asyncio backend not respecting the <code>PYTHONASYNCIODEBUG</code> environment variable when setting the <code>debug</code> flag in <code>anyio.run()</code> </li> <li> Fixed <code>SocketStream.receive()</code> not detecting EOF on asyncio if there is also data in the read buffer (<code>[#701](agronholm/anyio#701) <https://github.com/agronholm/anyio/issues/701></code>_) </li> <li> Fixed <code>MemoryObjectStream</code> dropping an item if the item is delivered to a recipient that is waiting to receive an item but has a cancellation pending (<code>[#728](agronholm/anyio#728) <https://github.com/agronholm/anyio/issues/728></code>_) </li> <li> Emit a <code>ResourceWarning</code> for <code>MemoryObjectReceiveStream</code> and <code>MemoryObjectSendStream</code> that were garbage collected without being closed (PR by Andrey Kazantcev) </li> <li> Fixed <code>MemoryObjectSendStream.send()</code> not raising <code>BrokenResourceError</code> when the last corresponding <code>MemoryObjectReceiveStream</code> is closed while waiting to send a falsey item (<code>[#731](agronholm/anyio#731) <https://github.com/agronholm/anyio/issues/731></code>_; PR by Ganden Schaffner) </li> </ul>  </blockquote> ... (truncated) </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/agronholm/anyio/commit/053e8f0a0f7b0f4a47a012eb5c6b1d9d84344e6a"><code>053e8f0</code></a> Bumped up the version</li> <li><a href="https://github.com/agronholm/anyio/commit/e7f750b96f5416d8ae932e15d726b5d03de80b67"><code>e7f750b</code></a> Fixed memory object stream sometimes dropping sent items (<a href="https://redirect.github.com/agronholm/anyio/issues/735">#735</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/9f5f14b3eb57f6965fc2c16879df93263bb020ea"><code>9f5f14b</code></a> Fixed task group getting cancelled if start() gets cancelled (<a href="https://redirect.github.com/agronholm/anyio/issues/717">#717</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/8b648bc213a85613b9441913b82a14d9cd839048"><code>8b648bc</code></a> Adjusted the pull request template</li> <li><a href="https://github.com/agronholm/anyio/commit/3ff5e9a6f1813152a7cc9ff27a8394a51812a040"><code>3ff5e9a</code></a> Rearranged changelog items</li> <li><a href="https://github.com/agronholm/anyio/commit/541d1f8197dfa36076f93b39e73ee5ad06012469"><code>541d1f8</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/agronholm/anyio/issues/734">#734</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/8a076900333b6b333f1748dd8d1e8ae8079a2924"><code>8a07690</code></a> Fix <code>MemoryObjectSendStream.send(falsey)</code> not raising <code>BrokenResourceError</code> w...</li> <li><a href="https://github.com/agronholm/anyio/commit/4b3de9737672df67b691f38543427e4869639f45"><code>4b3de97</code></a> Adjust the headings in the PR template</li> <li><a href="https://github.com/agronholm/anyio/commit/dfc44cf3c8c5444713258d0f1fda03e425240054"><code>dfc44cf</code></a> Added <code>__slots__</code> to <code>AsyncResource</code> (<a href="https://redirect.github.com/agronholm/anyio/issues/733">#733</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/96920b054c4d0c76ad440f36d7173ab5d5c86948"><code>96920b0</code></a> Fix typo in PR template (<a href="https://redirect.github.com/agronholm/anyio/issues/730">#730</a>)</li> <li>Additional commits viewable in <a href="https://github.com/agronholm/anyio/compare/4.3.0...4.4.0">compare view</a></li> </ul> </details> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=anyio&package-manager=pip&previous-version=4.3.0&new-version=4.4.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

dominik-schwabe added the bug Something isn't working label Mar 27, 2024

agronholm mentioned this issue Apr 4, 2024

Fix shield future to prevent erroneous RuntimeError #707

Merged

agronholm closed this as completed in #707 Apr 5, 2024

agronholm pushed a commit that referenced this issue Apr 5, 2024

Fix task_status future to prevent erroneous RuntimeError (#707)

e024e92

Fixes #706.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raising an error inside a task of a task group can produce: `RuntimeError: called 'started' twice on the same task status` #706

Raising an error inside a task of a task group can produce: `RuntimeError: called 'started' twice on the same task status` #706

dominik-schwabe commented Mar 27, 2024 •

edited

Loading

agronholm commented Mar 28, 2024

dominik-schwabe commented Mar 28, 2024

agronholm commented Mar 28, 2024

dominik-schwabe commented Mar 28, 2024

agronholm commented Mar 28, 2024

jonathanslenders commented Mar 28, 2024

davidbrochart commented Apr 3, 2024

Raising an error inside a task of a task group can produce: RuntimeError: called 'started' twice on the same task status #706

Raising an error inside a task of a task group can produce: RuntimeError: called 'started' twice on the same task status #706

Comments

dominik-schwabe commented Mar 27, 2024 • edited Loading

Things to check first

AnyIO version

Python version

What happened?

How can we reproduce the bug?

agronholm commented Mar 28, 2024

dominik-schwabe commented Mar 28, 2024

agronholm commented Mar 28, 2024

dominik-schwabe commented Mar 28, 2024

agronholm commented Mar 28, 2024

jonathanslenders commented Mar 28, 2024

davidbrochart commented Apr 3, 2024

Raising an error inside a task of a task group can produce: `RuntimeError: called 'started' twice on the same task status` #706

Raising an error inside a task of a task group can produce: `RuntimeError: called 'started' twice on the same task status` #706

dominik-schwabe commented Mar 27, 2024 •

edited

Loading