cpu install guide

[jiqing-feng]

This PR introduced a cpu install guide only for build from the source.

[jiqing-feng]

Hi @Titus-von-Koeller . Do you think we could introduce users to installing the CPU backend bnb? Even though it can only be built from source for now, it could encourage more users to use the multi-backend-refactor branch and help us keep making progress on that. WDYT?

[github-actions]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Titus-von-Koeller]

Hey @jiqing-feng,

yeah, totally, that's why I added this addition to the README and was just waiting for you all to get back to me about the right timing, so that we can all post on socials to recruit alpha testers.

Thanks a lot for adding these docs, that's a very helpful addition! The Intel backend should also be usable on Windows, right? Currently, it's only visible when having selected the Linux platform. You don't really see that when looking at the pure markdown, but the docs of the PR are live here.

However, I also think it would make more sense to have a separate section at the "###" level with descriptive title and adapt that also to the Windows section, if that's supported. We can then also link to that section directly and people more easily find it when scanning the page.

One other thing, could you please set up the pre-commit hooks as specified in our contributors guide and communicate that to the rest of your team as well? There were a few occasions where it was clear some of you don't have them enabled and it's an easy fix.

It's pretty cool, the lint actually correctly caught a typo in your commit :)

[Titus-von-Koeller]

cc @stevhliu as it's related to #1244 and it would be nice to come up with a coherent approach that's easy to understand for user's. Let me know if you have any thoughts, would be really appreciated 🤗

[stevhliu]

Copying over from my comment on the other PR :)

# Installation

bitsandbytes provides broad backend support for CUDA, AMD ROCm, Intel CPU/GPUs, and Apple silicon.

## CUDA
### Compile from source
### PyTorch CUDA versions

## AMD ROCm
install instructions for Linux/Windows
include pointer for rocm pitfall here

## Intel CPU + GPU
To install bitsandbytes on a CPU backend:

<installation code here>

## Apple Silicon (MPS)
install instructions for Linux/Windows

[jiqing-feng]

Hey @jiqing-feng,

yeah, totally, that's why I added this addition to the README and was just waiting for you all to get back to me about the right timing, so that we can all post on socials to recruit alpha testers.

Thanks a lot for adding these docs, that's a very helpful addition! The Intel backend should also be usable on Windows, right? Currently, it's only visible when having selected the Linux platform. You don't really see that when looking at the pure markdown, but the docs of the PR are live here.

However, I also think it would make more sense to have a separate section at the "###" level with descriptive title and adapt that also to the Windows section, if that's supported. We can then also link to that section directly and people more easily find it when scanning the page.

One other thing, could you please set up the pre-commit hooks as specified in our contributors guide and communicate that to the rest of your team as well? There were a few occasions where it was clear some of you don't have them enabled and it's an easy fix.

It's pretty cool, the lint actually correctly caught a typo in your commit :)

Hi @Titus-von-Koeller . Sorry for the delay, and thanks for your advice. I have fixed the code format, pls take a look.

Actually, we haven't tested it on Windows, we will check it and get feedback to you soon.

BTW, can I ask if you have any plans about merging multi-backend-refactor to the main?

[jiqing-feng]

Hi @Titus-von-Koeller . I am working on Windows support. When we talk about Windows, do you mean a desktop or a server or both?

BTW, I just notice that we actually don't have any C++ kernel in Intel CPU backend, so I didn't run cmake, and just pip install .. It works and the result is as expected, but shows an error log

ERROR:bitsandbytes.cextension:Could not load bitsandbytes native library: /home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared ob
ject file: No such file or directory
Traceback (most recent call last):
  File "/home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 124, in <module>
    lib = get_native_library()
  File "/home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 104, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
  File "/home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/jiqingfe/miniconda3/envs/bitsandbytes/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

The log has no impact on functionality and performance.

Do you think we could disable get_native_library when the backend is Intel CPU? It will make sense and be more user-friendly. Thx!

[Xia-Weiwen]

Hi @Titus-von-Koeller. We will need another PR to enable the int8 and 4bit path on Windows. We will enable the functionalities so that it won't break the commitment. However, performance is not guaranteed. Thanks.

[matthewdouglas]

Do you think we could disable get_native_library when the backend is Intel CPU? It will make sense and be more user-friendly. Thx!

@jiqing-feng There is the CPU implementation for F.quantize_blockwise in fp32 which written in C++, so it's looking for that. As of right now I think in this new CPU backend we throw a NotImplementedError, so what we could maybe do there is fall back to use the C++ implementation. Apart from that, I do expect that there will be changes coming to improve the UX. Specifically on this issue, I do think it could become each backend's responsibility to load the shared libraries it needs, and we would only initialize the backends we detect in the environment.

[jiqing-feng]

Do you think we could disable get_native_library when the backend is Intel CPU? It will make sense and be more user-friendly. Thx!

@jiqing-feng There is the CPU implementation for F.quantize_blockwise in fp32 which written in C++, so it's looking for that. As of right now I think in this new CPU backend we throw a NotImplementedError, so what we could maybe do there is fall back to use the C++ implementation. Apart from that, I do expect that there will be changes coming to improve the UX. Specifically on this issue, I do think it could become each backend's responsibility to load the shared libraries it needs, and we would only initialize the backends we detect in the environment.

I agree, so can we merge this PR first to show users how to enable the CPU backend under the Linux system or wait until we support the Windows system?

[jiqing-feng]

Hi @Titus-von-Koeller . I have added the Windows guide, but it requires the PR #1251 to be merged.

Have you got the machine from Intel? If so, we can help to set up the runner. It would be nice if you could share your email with me; my email is jiqing.feng@intel.com; pls send me an email if you have any concerns. Thx.

[Titus-von-Koeller]

Hey all! Sorry, I've been off sick the last 10 days unfortunately. I'll look into this as soon as I can, gotta get back up to speed with things.

[stevhliu]

Very nice, only a few suggestions to improve readability! 👍

[jiqing-feng]

Hi @Titus-von-Koeller @stevhliu . Please take a review for my new changes. Thx!

[Titus-von-Koeller]

thanks @jiqing-feng and @stevhliu, LGTM, really appreciated!