Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: support stream_options option #4967

Closed
NiuBlibing opened this issue May 22, 2024 · 1 comment · Fixed by #5135 or #5319
Closed

[Feature]: support stream_options option #4967

NiuBlibing opened this issue May 22, 2024 · 1 comment · Fixed by #5135 or #5319
Labels
feature request good first issue Good for newcomers

Comments

@NiuBlibing
Copy link
Contributor

🚀 The feature, motivation and pitch

According to openAI doc: https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options. The API provide the stream_options which can get token usage info for stream request.

Alternatives

No response

Additional context

No response
@simon-mo simon-mo added the good first issue Good for newcomers label May 22, 2024
@Etelis
Copy link
Contributor

Etelis commented May 24, 2024

I will implement this

Etelis added a commit to Etelis/vllm that referenced this issue May 30, 2024
@Etelis
[Feature]: Support stream_options option (vllm-project#4967)
9fb7aed 
- Introduced the `StreamOptions` class in `OpenAIBaseModel` with an optional `include_usage` attribute.

- Added `stream_options` attribute to the `ChatCompletionRequest` class, defaulting to `None`.
@Etelis Etelis mentioned this issue May 30, 2024
Merged
@Etelis Etelis mentioned this issue Jun 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
3 participants
@simon-mo @NiuBlibing @Etelis