run_process fails when running from .exe console_script entrypoint on windows

[graingert]

Things to check first

• I have searched the existing issues and didn't find my bug already reported there

• I have checked that my bug is still present in the latest release

AnyIO version

4.3.0

Python version

3.11.3

What happened?

running:

import anyio.to_process

from fastapi import FastAPI


app = FastAPI()

def sync_example():
    return 1

@app.get("/")
async def example() -> int:
    return await anyio.to_process.run_sync(sync_example)

with uvicorn demo:app fails with:

Traceback (most recent call last):
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 163, in run_sync
    await send_raw_command(pickled)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 89, in send_raw_command
    raise retval
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\User\\uvicron_demo\\Scripts\\uvicorn.exe\\__main__.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\User\uvicron_demo\Lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 412, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\errors.py", line 186, in __call__
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "C:\Users\User\uvicron_demo\Lib\site-packages\starlette\routing.py", line 74, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\fastapi\routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\demo.py", line 13, in example
    return await anyio.to_process.run_sync(sync_example)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\uvicron_demo\Lib\site-packages\anyio\to_process.py", line 168, in run_sync
    raise BrokenWorkerProcess(
anyio.BrokenWorkerProcess: Error during worker process initialization

How can we reproduce the bug?

see above

[agronholm]

Where should it get the interpreter path then?

[graingert]

sys.executable ?

[adilkhash]

I think the problem is more related to uvicorn rather than to anyio. Try using another asgi web server, for example hypercorn or run you app via main section like this:

if __name__ == '__main__':
    uvicorn.run('app:app', host="0.0.0.0", port=8000)

[agronholm]

As stated above, the problem is where to get the path to the interpreter, as sys.executable points to uvicorn.exe.

[adilkhash]

Then one of the workarounds might be using the following run command:

python -m uvicorn app:app

In this case sys.executable should point to python.exe

[agronholm]

Yes, that certainly should work.

[agronholm]

One thing that occurred to me: does multiprocessing still work from within a process launched as uvicorn? If so, where does it get the path to the actual Python interpreter?

[agronholm]

After digging a while, I saw in multiprocessing/spawn.py that it was doing os.path.join(sys.exec_prefix, 'python.exe'). Probably not a bad choice.

[agronholm]

I did some more digging and turns out, I was completely wrong about where the problem occurs. sys.executable was never the culprit. The problem was that it didn't know where to find the __main__ module. What I do there is getattr(sys.modules["__main__"], "__file__", None), but on Windows, that returns C:\...\uvicorn.exe\__main__.py! Even more bizarre is that if you try the same from the module itself, its __file__ attribute is completely fine.

[agronholm]

Right, I keep forgetting that the target function doesn't even live in the __main__ module, so there is no real point in trying to import that module. The subprocess end should check if the path to __main__ exists, and only then try to import it.

[agronholm]

Alright, this was a one-liner fix in the end.

[adilkhash]

Sounds cool, I can test it right away.

[agronholm]

I'm struggling a bit to write a failing test right now.

[agronholm]

I now understand why it only failed on Windows: __main__.__file__ points to a Python script bin/uvicorn on Linux/macOS, but on Windows, it points to an .exe wrapper which cannot be loaded by the Python import machinery. Thus I believe I need to point __main__.__file__ to an existing file that would fail to be imported.

[agronholm]

@adilkhash feel free to test against the linked PR.

[adilkhash]

it works! seems like the new release is coming 😉
thanks

[adilkhash]

looks like something is wrong, if I press Ctrl+C and KeyboardInterrupt is raised the following exception occurs:

Traceback (most recent call last):
  File "C:\...\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\...\Lib\asyncio\subprocess.py", line 72, in pipe_data_received
    reader.feed_data(data)
  File "C:\...\Lib\asyncio\streams.py", line 477, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof

[agronholm]

Can you share the script you tested with?

[adilkhash]

The one that is listed in this issue.
UPD: sorry, looks like it doesn't relate to that issue.

[agronholm]

Ok, so are you able to reproduce this other issue? Can you tell me how?

[adilkhash]

I am not sure if it is related to anyio at all.