bitsandbytes CUDA Errors After Manual Patch and Prior Working Install

[stephenholli]

Describe the bug

Starting yesterday I have been unable to get a functional instance running in 8bit or 4bit mode. Prior to performing a git pull on this repo yesterday I had a functional 4bit Llama instance working directly on Windows 10.

Now, after performing the bitsandbytes windows patch with libbitsandbytes_cuda116.dll and updating main.py, I still receive errors. As best I can tell, it appears to be trying to load the .so file instead of the dll. The following is the error text from my latest 1-click installer run.

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

1. Perform install on Windows 10 as per instructions in README.md.
2. Patch bitsandbytes per Support for LLaMA models #147 (comment)
3. Run server.py with gptq-bits 8

Screenshot

Logs

Starting the web UI...

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C')}
  warn(msg)
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: C:\Windows\System32\installer_files\env did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/Windows/System32/installer_files/env/etc/xml/catalog'), WindowsPath('/C'), WindowsPath('file')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
  warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Loading binary C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
[WinError 193] %1 is not a valid Win32 application
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: Loading binary C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
[WinError 193] %1 is not a valid Win32 application
C:\Windows\System32\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
No models are available! Please download at least one.
Press any key to continue . . .

System Info

Windows 10 - latest update
Nvidia RTX 4090 - latest driver - cuda toolkit 11.6

[oobabooga]

Duplicate of #400