[Bug]: Unable to run meta-llama/Llama-Guard-3-8B-INT8

[xfalcox]

Your current environment

Latest Docker image, RTX 4090

🐛 Describe the bug

docker run --gpus all vllm/vllm-openai:latest --model meta-llama/Llama-Guard-3-8B-INT8
...
[rank0]:     raise ValueError(f"Cannot find any of {keys} in the model's "
[rank0]: ValueError: Cannot find any of ['adapter_name_or_path'] in the model's quantization config.

[mgoin]

@thesues @chenqianfzh It looks like this is an 8bit BNB model. Would it be easy to add support for these checkpoints as well?

[chenqianfzh]

@thesues @chenqianfzh It looks like this is an 8bit BNB model. Would it be easy to add support for these checkpoints as well?

It won't be difficult. I will work on it with higher priority.

[meihui]

seems version 0.5.4+cu124 is working with bnb 4bit model.

but it says

WARNING 08-06 06:27:07 config.py:254] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.

Will that be a easy fix/support too?

[chenqianfzh]

#7445

with this PR, meta-llama/Llama-Guard-3-8B-INT8 is supported.

[chenqianfzh]

The speed can be slower than

A lot of quantizations here are not in the optimized method list yet. It is not our top priority now to optimize the speed yet, as we are working to support more quantization features of bnb.