Anyio version 3 breaks AsyncContentsManager

[kevin-bates]

Description

Anyio version 3 was recently introduced and we found that the various AsyncContentsManager calls that use anyio's run_sync_in_worker_thread method fail with a traceback similar to:

HTTPServerRequest(protocol='http', host='127.0.0.1:38403', method='GET', uri='/a%40b/api/contents/Directory%20with%20spaces%20in', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 697, in run_sync_in_worker_thread
    work_queue = _threadpool_work_queue.get()
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/lowlevel.py", line 120, in get
    raise LookupError(f'Run variable "{self._name}" has no value and no default set')
LookupError: Run variable "_threadpool_work_queue" has no value and no default set

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/home/runner/work/jupyter_server/jupyter_server/jupyter_server/services/contents/handlers.py", line 111, in get
    path=path, type=type, format=format, content=content,
  File "/home/runner/work/jupyter_server/jupyter_server/jupyter_server/utils.py", line 176, in ensure_async
    result = await obj
  File "/home/runner/work/jupyter_server/jupyter_server/jupyter_server/services/contents/filemanager.py", line 692, in get
    model = await self._dir_model(path, content=content)
  File "/home/runner/work/jupyter_server/jupyter_server/jupyter_server/services/contents/filemanager.py", line 574, in _dir_model
    dir_contents = await run_sync_in_worker_thread(os.listdir, os_dir)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/to_thread.py", line 37, in run_sync_in_worker_thread
    return await run_sync(func, *args, cancellable=cancellable, limiter=limiter)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/to_thread.py", line 29, in run_sync
    limiter=limiter)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 707, in run_sync_in_worker_thread
    find_root_task().add_done_callback(_loop_shutdown_callback)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 56, in find_root_task
Error:     raise RuntimeError('Cannot find root task for setting cleanup callback')

The failure was also seen on a build of PR 483 prior to capping anyio < 3.

Reproduce

1. From the jupyter_server repo
2. Ensure anyio version 3 is installed (released on April 20)
3. Run pytest -v jupyter_server/tests/services/contents/test_largefilemanager.py::test_save
4. See failure similar to above.

Expected behavior

Tests should pass.

Context

• Operating System and version: MacOS/Ubuntu (not sure about Windows)
• Browser and version: N/A
• Jupyter Server version: master

I noticed the following deprecation warning when running pytest:

jupyter_server/tests/services/contents/test_largefilemanager.py::test_save[AsyncLargeFileManager]
  /opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/to_thread.py:36: DeprecationWarning: run_sync_in_worker_thread() has been deprecated, use anyio.to_thread.run_sync() instead
    DeprecationWarning)

However, a quick attempt to switch to anyio.to_thread.run_sync() resulted in the same issue since it appears run_sync() still calls run_sync_in_worker_thread() internally. As a result, this may be an anyio issue.

Using run_sync()

ERROR    traitlets:filemanager.py:746 Error while saving file: Untitled.ipynb [Errno 2] No such file or directory: '/private/var/folders/4w/qxgsbhyx1y93qlk9mnzwjryc0000gn/T/pytest-of-kbates/pytest-74/test_save_AsyncLargeFileManage0/.~Untitled.ipynb' -> '/private/var/folders/4w/qxgsbhyx1y93qlk9mnzwjryc0000gn/T/pytest-of-kbates/pytest-74/test_save_AsyncLargeFileManage0/Untitled.ipynb'
Traceback (most recent call last):
  File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 697, in run_sync_in_worker_thread
    work_queue = _threadpool_work_queue.get()
  File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/lowlevel.py", line 120, in get
    raise LookupError(f'Run variable "{self._name}" has no value and no default set')
LookupError: Run variable "_threadpool_work_queue" has no value and no default set
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 121, in atomic_writing

yield fileobj

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 218, in atomic_writing

yield f

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 389, in _save_notebook

await run_sync(partial(nbformat.write, version=nbformat.NO_CONVERT), nb, f)

File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/to_thread.py", line 29, in run_sync

limiter=limiter)

File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 707, in run_sync_in_worker_thread

find_root_task().add_done_callback(_loop_shutdown_callback)

File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 56, in find_root_task

raise RuntimeError('Cannot find root task for setting cleanup callback')

RuntimeError: Cannot find root task for setting cleanup callback
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/filemanager.py", line 732, in save

await self._save_notebook(os_path, nb)

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 389, in _save_notebook

await run_sync(partial(nbformat.write, version=nbformat.NO_CONVERT), nb, f)

File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/contextlib.py", line 130, in exit

self.gen.throw(type, value, traceback)

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 218, in atomic_writing

yield f

File "/opt/anaconda3/envs/jupyter-server-dev/lib/python3.7/contextlib.py", line 130, in exit

self.gen.throw(type, value, traceback)

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 125, in atomic_writing

replace_file(tmp_path, path)

File "/Users/kbates/repos/oss/jupyter/jupyter_server/jupyter_server/services/contents/fileio.py", line 35, in replace_file

os.replace(src, dst)

FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/4w/qxgsbhyx1y93qlk9mnzwjryc0000gn/T/pytest-of-kbates/pytest-74/test_save_AsyncLargeFileManage0/.~Untitled.ipynb' -> '/private/var/folders/4w/qxgsbhyx1y93qlk9mnzwjryc0000gn/T/pytest-of-kbates/pytest-74/test_save_AsyncLargeFileManage0/Untitled.ipynb'

cc: @mwakaba2

[mwakaba2]

Hi @kevin-bates ! thanks for the detailed bug report. I can take a look and try to get a fix out end of this week. Is that ok?

[kevin-bates]

No worries. The dependency is capped so we're good right now. Whenever you can poke at this is fine - thank you!

[mwakaba2]

Hi @kevin-bates, I looked into this a bit today, and it seems like there could be a bug in anyio, so I created a github issue for that and proposed a fix. I hope we get an answer soon!