Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Turn shareGPT data into a standard benchmark #45

Closed
zhuohan123 opened this issue Apr 22, 2023 · 0 comments · Fixed by #145
Closed

Turn shareGPT data into a standard benchmark #45

zhuohan123 opened this issue Apr 22, 2023 · 0 comments · Fixed by #145
Assignees
@zhuohan123

Comments

@zhuohan123
Copy link
Member

  1. Extract out the lengths of the conversation rounds, and maybe have that data directly available from github.
  2. The current L-shape evaluation with binary search for throughput is hard to run and not scalable. We should find an easier way to benchmark the performance.
@WoosukKwon WoosukKwon mentioned this issue Jun 11, 2023
Merged
@shanshanpt shanshanpt mentioned this issue Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this issue Nov 20, 2023
Closed
tianyil1 pushed a commit to tianyil1/vllm that referenced this issue Jun 5, 2024
@kzawora-intel
WA: Remove pyproject.toml, bypass HPU autodetection (vllm-project#45)
8359489
fxmarty pushed a commit to fxmarty/vllm-public that referenced this issue Jun 12, 2024
@mawong-amd
Include benchmark scripts in container (vllm-project#45)
95b3acc
yukavio pushed a commit to yukavio/vllm that referenced this issue Jul 3, 2024
@mgoin
Update arg_utils.py with semi_structured_sparse_w16a16 (vllm-projec…
757e48a 
…t#45)

Tested by checking the help message in openai server:
```
python -m vllm.entrypoints.openai.api_server --help
```

Before:
```
  --sparsity {sparse_w16a16,None}, -s {sparse_w16a16,None}
                        Method used to compress sparse weights. If None, we first check the `sparsity_config`
                        attribute in the model config file. If that is None we assume the model weights are dense
 ```
 
 After:
```
--sparsity {None,sparse_w16a16,semi_structured_sparse_w16a16}, -s
{None,sparse_w16a16,semi_structured_sparse_w16a16}
Method used to compress sparse weights. If None, we first check the
`sparsity_config`
attribute in the model config file. If that is None we assume the model
weights are dense
```
@ZHJ19970917 ZHJ19970917 mentioned this issue Jul 14, 2024
Closed
@qniguogym qniguogym mentioned this issue Jul 29, 2024
Merged
3 tasks
@alixiaodi alixiaodi mentioned this issue Aug 2, 2024
Closed
jikunshang pushed a commit to jikunshang/vllm that referenced this issue Aug 19, 2024
@yma11 @jikunshang
update docker file with dependency installation (vllm-project#45)
e9e3a5c
jikunshang pushed a commit to jikunshang/vllm that referenced this issue Sep 11, 2024
@yma11 @jikunshang
update docker file with dependency installation (vllm-project#45)
9ff9708
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuohan123 zhuohan123

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add script for benchmarking serving throughput vllm-project/vllm
1 participant
@zhuohan123