Using bitsandbytes on Windows 11

To load large models in quantized form you need the bitsandbytes library, but it was originally designed to be installed and used under Linux. So to use it on Windows 11 you need to make a few changes.

After pip install bitsandbytes, modify the source in two places.

With conda installed, the directory is usually: {driver}:\conda\envs\{name}\Lib\site-packages\bitsandbytes\. pip install usually prints the install location — just copy it.

First, in that directory, copy the downloaded libbitsandbytes_cuda116.dll file into it.

The main changes are in two files. In cuda/main.py, the evaluate_cuda_setup() function can be made to return "libbitsandbytes_cuda116.dll" directly, because the original code loads the Linux .so file by default — so it must be changed to the corresponding DLL.

The second place is cextension.py in the same directory tree: change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path)).

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))