Skip to content

Commit

Permalink
Revert "fix merge_async_iterators usage for vLLM>0.5.4"
Browse files Browse the repository at this point in the history
This reverts commit 09290e6.

The mandatory `is_cancelled` kwarg was made optional in
vllm-project/vllm#7282
  • Loading branch information
dtrifiro committed Aug 8, 2024
1 parent 96b8e0f commit 455bee8
Showing 1 changed file with 3 additions and 12 deletions.
15 changes: 3 additions & 12 deletions src/vllm_tgis_adapter/grpc/grpc_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,18 +255,9 @@ async def Generate(
)

# TODO handle cancellation
result_generator: AsyncIterator[tuple[int, RequestOutput]]

kwargs = {}
if "is_cancelled" in inspect.signature(merge_async_iterators).parameters:
# vllm > 0.5.4

async def is_cancelled() -> bool:
return context.cancelled()

kwargs["is_cancelled"] = is_cancelled

result_generator = merge_async_iterators(*generators, **kwargs)
result_generator: AsyncIterator[tuple[int, RequestOutput]] = (
merge_async_iterators(*generators)
)

resp_options = request.params.response
responses: list = [None] * request_count
Expand Down

0 comments on commit 455bee8

Please sign in to comment.