can not get UI running - "The installed version of bitsandbytes was compiled without GPU support"

[simingzhao]

Describe the bug

this is the error message:

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

python3 server.py

Screenshot

Logs

Gradio HTTP request redirected to localhost :)
bin /Users/siming/anaconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/Users/siming/anaconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "

System Info

Mac M1 Pro

[cornpo]

bitsandbytes-foundation/bitsandbytes#252

[Manoz]

You have a different issue. The error with bitsandbytes is just a warning. I have the same warning on my Mac and everything works perfectly. The real issue is the KeyError: 'serialized_input'. It has been solved here.

You might want to pull the latest changes on the repository and do a pip install -r requirements.txt --upgrade to update the libraries.

If you're still struggling, here's the setup I have with libraries versions on my Apple Silicon M2.

[silvacarl2]

try this:

bitsandbytes debugging in case no GPU error:

pip uninstall bitsandbytes && pip install bitsandbytes
python -m bitsandbytes && python -m torch.utils.collect_env

if you still get the error, this is a hack that fixes it, at least on WSL:

cp /home/silvacarl/.local/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cuda117.so /home/silvacarl/.local/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so

[milezzz]

I have the same exact issue and hardware as OP and nothing in this thread helped. Has anyone on M1/M2 got this working properly? If so which model please?

[cornpo]

@milezzz Have you tried pip install --upgrade git+https://github.com/TimDettmers/bitsandbytes.git

[milezzz]

Thanks @cornpo I upgraded bitsandbytes but got the same error:

bin /Users/milezzz/miniforge3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/Users/milezzz/miniforge3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
INFO:Loading MetaIX_GPT4-X-Alpaca-30B-4bit...
INFO:llama.cpp weights detected: models/MetaIX_GPT4-X-Alpaca-30B-4bit/gpt4-x-alpaca-30b-ggml-q4_1.bin

Does it matter which model I am testing with?

[simingzhao]

pip install --upgrade git+https://github.com/TimDettmers/bitsandbytes.git

still not working.....displaying the same error message

[simingzhao]

@milezzz Have you tried pip install --upgrade git+https://github.com/TimDettmers/bitsandbytes.git

still the same error, does it related to the model I'm testing?

[github-actions]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.