Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Commit

Permalink
[BugFix] Ensure worker model loop is always stopped at the right time (
Browse files Browse the repository at this point in the history
  • Loading branch information
njhill authored and robertgshaw2-neuralmagic committed Jul 7, 2024
1 parent 3555aff commit ef17c9b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion vllm/engine/llm_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -840,7 +840,7 @@ def step(self) -> List[Union[RequestOutput, EmbeddingRequestOutput]]:
# Tracing
self.do_tracing(scheduler_outputs)

if not request_outputs:
if not self.has_unfinished_requests():
# Stop the execute model loop in parallel workers until there are
# more requests to process. This avoids waiting indefinitely in
# torch.distributed ops which may otherwise timeout, and unblocks
Expand Down

0 comments on commit ef17c9b

Please sign in to comment.