Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RHOAIENG-13027: Adding note for chat_template requirement #455

Merged
merged 5 commits into from
Sep 18, 2024

Conversation

syaseen-rh
Copy link
Contributor

@syaseen-rh syaseen-rh commented Sep 17, 2024

Description

Adding note for chat_template requirement (RHOAIENG-12233)

As of vLLM v0.5.5, you must provide a chat template while querying a model using the /v1/chat/completions endpoint. If your model does not include a predefined chat template, you can use the chat-template command-line parameter to specify a chat template in your custom vLLM runtime, as shown in the example. Replace <CHAT_TEMPLATE> with the path to your template.

containers:
  - args:
      - --chat-template=<CHAT_TEMPLATE>
You can use the chat templates that are available as `.jinja` files link:https://github.com/opendatahub-io/vllm/tree/main/examples[here] or with the vLLM image under `/apps/data/template`. For more information about chat templates, see link:https://huggingface.co/docs/transformers/main/chat_templating[Chat templates].

How Has This Been Tested?

Local build

Upstream preview:

Screenshot 2024-09-18 at 11 14 32 AM

Downstream preview:
Screenshot 2024-09-18 at 11 16 29 AM

@aduquett
Copy link
Contributor

LGTM

@RH-steve-grubb
Copy link

Looks OK to me.

@syaseen-rh syaseen-rh merged commit f62ac58 into opendatahub-io:main Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants