add vlmm backend #274

NathanHB · 2024-08-21T13:21:34Z

what this PR does:

adds vllm as backend for faster inference.

how to use:

 lighteval accelerate --model_args="pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,dtype=bfloat16,vllm,data_parallel_size=2" use_chat_template --tasks "leaderboard|arc:challenge|0|0,extended|ifeval|0|0,lighteval|gsm8k|5|1" output_dir="./evals/"

generate untill
loglikelihood
multiple generation metrics

src/lighteval/models/vllm_model.py

src/lighteval/data.py

src/lighteval/models/model_config.py

src/lighteval/models/model_loader.py

src/lighteval/models/vllm_model.py

Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>

…l into nathan-vllm-backend

add vlmm backend

06ead43

clefourrier reviewed Aug 21, 2024

View reviewed changes

src/lighteval/models/vllm_model.py Show resolved Hide resolved

clefourrier approved these changes Aug 22, 2024

View reviewed changes

NathanHB and others added 13 commits August 26, 2024 12:53

add vlmm backend

ea0cd3d

Update src/lighteval/models/model_config.py

0a66f40

Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>

Merge branch 'nathan-vllm-backend' of github.com:huggingface/lighteva…

6dc35c6

…l into nathan-vllm-backend

fix from review

8b2392a

fix

9029943

fix

5c9029c

fix

d0b46e9

fix

97632d9

fix

fe8da51

fix

b52239d

Merge branch 'main' into nathan-vllm-backend

1b57b26

fix

ed7b35d

remove unescerary print

b756f4c

NathanHB merged commit 21934d5 into main Sep 3, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add vlmm backend #274

add vlmm backend #274

NathanHB commented Aug 21, 2024 •

edited

Loading

add vlmm backend #274

add vlmm backend #274

Conversation

NathanHB commented Aug 21, 2024 • edited Loading

NathanHB commented Aug 21, 2024 •

edited

Loading