We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lora_dtype
Versions of relevant libraries: [pip3] numpy==1.22.4 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] torch==2.3.0 [pip3] transformers==4.41.1 [pip3] triton==2.3.0 [pip3] vllm-nccl-cu12==2.18.1.0.4.0 [conda] numpy 1.22.4 py39hc58783e_0 conda-forge [conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi [conda] torch 2.3.0 pypi_0 pypi [conda] transformers 4.41.1 pypi_0 pypi [conda] triton 2.3.0 pypi_0 pypi [conda] vllm-nccl-cu12 2.18.1.0.4.0 pypi_0 pypi ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.4.2
import vllm vllm.EngineArgs(model='a', lora_dtype='float16')
raises
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[40], line 1 ----> 1 ea = vllm.EngineArgs(model='a', lora_dtype='float16') TypeError: __init__() got an unexpected keyword argument 'lora_dtype'
Modify lora_dtype = 'auto' to lora_dtype: str = 'auto' in arg_utils.py.
lora_dtype = 'auto'
lora_dtype: str = 'auto'
The text was updated successfully, but these errors were encountered:
Successfully merging a pull request may close this issue.
Your current environment
🐛 Describe the bug
raises
Fix
Modify
lora_dtype = 'auto'
tolora_dtype: str = 'auto'
in arg_utils.py.The text was updated successfully, but these errors were encountered: