Socket context #393
Socket context #393
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
| async def _send_one_way_rpc_request(self, request: RPC_REQUEST_TYPE, | ||
| error_message: str): | ||
| """Send one-way RPC request to trigger an action.""" | ||
| @contextmanager |
There was a problem hiding this comment.
does this need to be asynccontextmanager?
There was a problem hiding this comment.
I don't think so because it doesn't use any async methods or do any blocking
There was a problem hiding this comment.
Based on this I think it's just poll / send / recv that need to be handled with async:
https://pyzmq.readthedocs.io/en/latest/api/zmq.asyncio.html#socket
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
joerunde
merged commit 0b351c0
into
neuralmagic:isolate-oai-server-process
Ensures no sockets are leaked on the client-side
Also postpones the server shutdown await so that the backend can shutdown concurrently, and all connections can be cleaned up at the same time. This prevents hangs where the frontend blocks on remaining connections but the backend has not yet initiated shutdown