AttributeError: ‘NoneType’ object has no attribute ‘Llama’

[sleepingforest1024]

I am currently executing text-generation-webui in a VM environment, using Anaconda, and have successfully GIT PULLED before initializing the server on http://127.0.0.1:7860/.

However, I am encountering an error when attempting to run codellama-13b-python.Q5_0.gguf or any similar variations with llama.cpp loader. Below is the traceback for your reference:

Traceback (most recent call last):

File “C:\python\text-generation-webui\modules\ui_model_menu.py”, line 201, in load_model_wrapper

shared.model, shared.tokenizer = load_model(shared.model_name, loader)
File “C:\python\text-generation-webui\modules\[models.py](http://models.py/)”, line 78, in load_model

output = load_func_map[loader](model_name)
File “C:\python\text-generation-webui\modules\[models.py](http://models.py/)”, line 224, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)
File “C:\python\text-generation-webui\modules\llamacpp_model.py”, line 54, in from_pretrained

Llama = llama_cpp_lib().Llama
AttributeError: ‘NoneType’ object has no attribute ‘Llama’

May I inquire if anyone has faced a similar issue and if there are any known solutions or workarounds for this problem?

[kaneda2004]

Ya I just had the same issue.

Solved it on my machine (macos sonoma 14.0) by adding this line:

https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/metal/llama_cpp_python-0.2.7-cp310-cp310-macosx_13_0_arm64.whl; platform_system == "Darwin" and platform_release >= "23.0.0"

to the requirements_apple_silicon.txt file

then enter bash update_macos.sh in your working dir

Once it installs the new version of llama.cpp you should be good to go

[benderis]

I have exact same problem but run text-generation-webui through Pinokio. Is there any way to install this fix ? I update requirements_apple_silicon.txt and run update in Pinokio but I think it didn't update anything.

My hardware apple m2 and run macos sonoma

[Euplectello]

Hello all,

I get the same error, regardless of the model I use. I use Windows and had the chat running after installing with "start_windows.bat".
When I closed the web UI and restarted with "python server.py" I get the errora as shown below.
However, if I run "start_windows.bat" I do not get those errors.
Any ideas or suggestions ?

Thank you!

2023-09-29 17:16:17 INFO:Loading llama-2-7b-chat.Q2_K.gguf...
2023-09-29 17:16:17 INFO:llama.cpp weights detected: models\llama-2-7b-chat.Q2_K.gguf
2023-09-29 17:16:17 ERROR:Failed to load the model.
Traceback (most recent call last):
File "D:\Betazed\pythonProjects\text-generation-webui\modules\ui_model_menu.py", line 201, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Betazed\pythonProjects\text-generation-webui\modules\models.py", line 78, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Betazed\pythonProjects\text-generation-webui\modules\models.py", line 224, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Betazed\pythonProjects\text-generation-webui\modules\llamacpp_model.py", line 54, in from_pretrained
Llama = llama_cpp_lib().Llama
^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'Llama'

[kaneda2004]

To everyone replying that they have the same problem, you can fix it by repeating the steps I set out originally, but by replacing the .whl file with the one most appropriate for your environment -- in my case it was arm64 w/ most recent macos, if you have windows and inference is on cpu, then grab the correct url for your env: Here
Here's all the flavors:
Here

[cyetman]

Dumb question here - how do I know which wheel file is most appropriate for my env. ?

[Falenos]

Regarding this, I had the same issue, did the steps mentioned by @kaneda2004 (thanks!) and things finally worked for GGUF models.

However, I digged a bit deeper and realised that apparently I did not have a good system regarding python for m1 and pytorch that allows cublas. This was what lead to requirements_apple_silicon.txt and some .sh scripts failure, and installing different metal wheels (.whl) file.
My previous infra was miniconda installed with brew.

If you're using a Mac with an M1 chip, it's recommended to use Miniforge instead of Miniconda. The reason is that Miniforge is a variant of Miniconda that is designed to support conda-forge, a community-led collection of recipes, build infrastructure, and distributions for the conda package manager. Conda-forge provides binary conda packages for a wide range of software, built to be highly compatible across platforms and systems. This includes support for the arm64 architecture used in Apple's M1 chips docs.conda.io.

While it's possible to install Miniconda with brew and then add the conda-forge channel, Miniforge comes with conda-forge set up as the default channel, which makes it easier to install packages that are compatible with the M1 chip stackoverflow.com.

So instead of miniconda from brew, I scrapped miniconda, scrapped python related stuff from brew (except 3.11 in my case that is used by a bunch of tools, running brew uninstal [whatever] will tell you which), scrapped all relevant paths from ~/.zshrc and did a miniforge3 installation using curl.

After that I did not reinstalled the webgui just pulled, so I was just checking what other stuff i needed to reinstall from the main README

For the installation of pytorch I used nightly the the command from the website

Installation of llama-cpp-python is important, following the instructions for macos
https://github.com/abetlen/llama-cpp-python#installation-from-pypi
https://github.com/abetlen/llama-cpp-python/blob/main/docs/install/macos.md

I pulled the latest changes

I run pip install -r requirements.txt

Now there is gpu acceleration for my models, which improved the performance by 4x.

Not an expert on these stuff, but this is what i did and i am happy with the outcome

NOTE:
the dependency GRadio (for ui components in webui, I think) was throwing errors but by running
pip install -r requirements.txt --upgrade and upgrading I have no issues a.t.m.

[goranapivis]

Tried this, but still got the User warning:

/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'

[knightcn1983]

this is could solve problem

[mega-ice]

Hi, got the same error after i updated several model backends. After a while i realised i didnt update llama-cpp-python properly because it uses submodule link to original llama.cpp.
So when updating, do not forget the --recurse-submodules part.
In your llama-cpp-python directory (and environment) run

git fetch && git pull --recurse-submodules

and then compile your gpu flavour :) I use the latest AMD ROCM 5.7.1 and PyTorch Nightly, (RDNA3 architecture), HIPBLAS.

CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DAMDGPU_TARGETS=gfx1100" FORCE_CMAKE=1 pip install . --upgrade --no-cache-dir

After that everything works fine.

[goranapivis]

Running on M1 with Ventura 13.6

ggml_metal_graph_compute: command buffer 0 failed with status 5
GGML_ASSERT: /Users/runner/work/llama-cpp-python-cuBLAS-wheels/llama-cpp-python-cuBLAS-wheels/vendor/llama.cpp/ggml-metal.m:1369: false
/bin/sh: line 1: 80242 Abort trap: 6 python server.py

[goranapivis]

Got LM Studio working with GPU support, what a big difference in speed compared to this! :-)

[DdrAGgoNn]

AttributeError: 'NoneType' object has no attribute 'Llama'
everyone is saying run this run that, to be honest I don't know where to run those code...
I am using a MacBook Pro 2017 with i5 8G ram, and I was trying to run mistral 7B model, I thought it would work cause I have 8G of ram.
it didn't.
how?

[aaron13100]

Me too.

ERROR:Failed to load the model.
Traceback (most recent call last):
  File "/Users/user/Downloads/text-generation-webui/modules/ui_model_menu.py", line 209, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(shared.model_name, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/text-generation-webui/modules/models.py", line 85, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/text-generation-webui/modules/models.py", line 250, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/text-generation-webui/modules/llamacpp_model.py", line 54, in from_pretrained
    Llama = llama_cpp_lib().Llama
            ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'Llama'

I tried adding https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/cpu/llama_cpp_python-0.2.20+cpubasic-cp311-cp311-manylinux_2_31_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.11" and various others to requirements_cpu_only.txt and running pip install -r requirements_cpu_only.txt , but the result is the same. I don't know what to try next.

[cahya12]

same with me, i've running pip install -r requirements_cpu_only.txt. but still the same. need help please

[burrizza]

The error probably means that the llama_cpp module could not be imported by the script modules/llamacpp_model.py
You can maybe get more useful information if you try to load the library without catching the exception. Simply change the lines of that file to

import llama_cpp
"""
try:
    import llama_cpp
except:
    llama_cpp = None
"""

and try to load a model again. The new error message should be more specific which should make it easier to get help.

In my case there were path variables set to a missing ROCm folder, which caused llama_cpp to not load properly.
Good luck!

See also #4817

[Mero7979]

I installed text-generation-webui.
during installation, i chose N) none (CPU mode)

I had previously installed Pytorch with this configuration:

When I load the model I get this error: AttributeError: ‘NoneType’ object has no attribute ‘Llama’

I tried the solution proposed by kaneda2004. Your solution is not understandable to me. I am not an expert on these issues.

Please, someone who can make a tutorial to solve the problem. I'm not familiar with this and need step by step help.

My configurations:
PC = Windows 10 x64
CPU = Intel(R) Xeon(R) CPU W3570 @ 3.20GHz || Instruction Set Extensions >> Intel® SSE4.2 (I think dont have AVX or AVX2 or AVX512)
Conda = 24.2.1
Torch = 2.2.1
Python = 3.11.7

Thanks for the support.

[thistleknot]

I too am having this issue after updating to the latest text-generation-webui

If I had to guess, I suspect it's llama.cpp

``
Traceback (most recent call last):

File "/data/text-generation-webui/modules/ui_model_menu.py", line 249, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)
File "/data/text-generation-webui/modules/models.py", line 94, in load_model

output = load_func_maploader
File "/data/text-generation-webui/modules/models.py", line 272, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)
File "/data/text-generation-webui/modules/llamacpp_model.py", line 63, in from_pretrained

Llama = llama_cpp_lib().Llama
AttributeError: 'NoneType' object has no attribute 'Llama'

CMAKE_ARGS="-DLLAMA_CUBLAS=on -DLLAMA_AVX=on -DLLAMA_AVX2=off -DBUILD_SHARED_LIBS" pip install -r requirements_noavx2.txt

[thistleknot]

likely compute < 8.0
I suspect it's llama-cpp-python, as when I attempted to load model using that library itself, glibc error

so I resorted to my tried and true installing llama.cpp manually after
        git clone https://github.com/ggerganov/llama.cpp
        cd llama.cpp
        rm -rf build; mkdir build;cd build;cmake ..;CMAKE_ARGS="-DLLAMA_CUBLAS=on -DLLAMA_AVX=on -DLLAMA_AVX2=off -DBUILD_SHARED_LIBS" cmake --build . --config Release;cd ..;LLAMA_CUBLAS=1 make libllama.so

rm -rf llama-cpp-python; git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git; cd llama-cpp-python/;CMAKE_ARGS="-DLLAMA_CUDA=on -DLLAMA_AVX=on -DLLAMA_AVX2=off" pip install -e .[all] --force

then commented out all llama-* in the noavx2 requirements, then installed

[amanjagdev]

I went to the repo mentioned by @kaneda2004 because the command that they mentioned didn't work well for me.

You can try running python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/basic/cpu command.

Easiest way to run this in web ui's environment is to add this in any function present in one_click.py.

    if is_macos():
        print_big_message("Starting Installation of llama-cpp for MacOS.")
        run_cmd("python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/basic/cpu", assert_success=True, environment=True)
        print_big_message("Ending Installation of llama-cpp for MacOS.")

I added it in def update_pytorch():
and then run update_wizard_macos.sh and chose option A

And it worked for me!

[LoganMartinez]

For MacOS users with Sequoia, the wheels linked above doesn't have a version for OS 15 yet. I was able to get it working with the wheels listed on llama.cpp's pypI page, which doesn't seem to be specific to a particular version. Here's what my requirements.txt looks like:

# Mac wheels
--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal
llama-cpp-python