Skip to content

Commit

Permalink
fix: enable logprobs during spec decoding by default
Browse files Browse the repository at this point in the history
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
  • Loading branch information
tjohnson31415 committed Aug 20, 2024
1 parent 19adb9d commit 479d393
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions Dockerfile.ubi
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,11 @@ RUN --mount=type=cache,target=/root/.cache/pip \
ENV HF_HUB_OFFLINE=1 \
PORT=8000 \
HOME=/home/vllm \
# As an optmization, vLLM disables logprobs when using spec decoding by
# default, but this would be unexpected to users of a hosted model that
# happens to have spec decoding
# see: https://github.com/vllm-project/vllm/pull/6485
DISABLE_LOGPROBS_DURING_SPEC_DECODING=false \
# Allow requested max length to exceed what is extracted from the
# config.json
# see: https://github.com/vllm-project/vllm/pull/7080
Expand Down

0 comments on commit 479d393

Please sign in to comment.