Architecture not yet supported for local LLM inference via `Oobabooga` #636

ashakiro · 2023-10-13T02:19:28Z

Describe the bug

windows 11
i'm trying to run local model
interpreter --local --model /TheBloke/CodeLlama-34B-Instruct-GGUF
After model has been downloaded it stucked on:
Getting started...
Ctrl+C
setup_text_llm.py", line 56, in setup_text_llm
raise Exception("Architecture not yet supported for local LLM inference via Oobabooga. Please run interpreter to connect to a cloud model.")
Exception: Architecture not yet supported for local LLM inference via Oobabooga. Please run interpreter to connect to a cloud model.
How can be fixed?

Reproduce

interpreter --local --model /TheBloke/CodeLlama-34B-Instruct-GGUF

Expected behavior

open interpreter

Screenshots

Open Interpreter version

0.1.9

Python version

3.11.5

Operating System name and version

windows 11

Additional context

No response

The text was updated successfully, but these errors were encountered:

WonkySoft · 2023-10-13T20:49:14Z

I hit this issue after updating today. From what I can tell Oobabooga is a webui. I don't want a UI, sigh.

Kwalladee · 2023-10-14T04:51:49Z

I am having the same issue starting interpreter with --local but on the default Mistral 7b for the newest commit. Stuck on 'getting started...'

sebvannistel · 2023-10-14T07:41:06Z

exactly the same for me too

Notnaton · 2023-10-14T09:15:06Z

Hi, I have the same problem.

OS: Windows 10
CPU: AMD Ryzen 7 3700X 8-Core Processor
RAM: 64,0 GB
GPU: AMD Radeon RX 5700 XT
Python: Python 3.11.0

Previous working Open-interpreter versions for me: 0.1.3 and 0.1.7

I ran pip install --upgrade open-interpreter

interpreter --local

C:\Users\NAME>interpreter --local

Open Interpreter will use Mistral 7B for local execution.

▌ Warning: Local LLM usage is an experimental, unstable feature.

Located 'C:\Users\NAME\AppData\Local\ooba\ooba\models\mistral-7b-instruct-v0.1.Q4_0.gguf', which has already been downloaded.

Getting started...

Hitting ctrl + c because it is not doing anything after "Getting started..."

Error:

Traceback (most recent call last):
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_text_llm.py", line 45, in setup_text_llm
return setup_local_text_llm(interpreter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_local_text_llm.py", line 40, in setup_local_text_llm
ooba_llm = ooba.llm(path, verbose=interpreter.debug_mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\ooba\llm.py", line 51, in init
open_ports = get_open_ports(2000, 10000)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\ooba\utils\get_open_ports.py", line 8, in get_open_ports
result = s.connect_ex((host, port)) # returns 0 if the connection was successful
^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

▌ Failed to install huggingface/TheBloke/Mistral-7B-Instruct-v0.1-GGUF.

We have likely not built the proper huggingface/TheBloke/Mistral-7B-Instruct-v0.1-GGUF support for your system.

(Running language models locally is a difficult task! If you have insight into the best way to implement this across platforms/architectures, please join the Open Interpreter community Discord, or the Oobabooga community Discord, and
consider contributing the development of these projects.)

Traceback (most recent call last):
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_text_llm.py", line 45, in setup_text_llm
return setup_local_text_llm(interpreter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_local_text_llm.py", line 40, in setup_local_text_llm
ooba_llm = ooba.llm(path, verbose=interpreter.debug_mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\ooba\llm.py", line 51, in init
open_ports = get_open_ports(2000, 10000)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\ooba\utils\get_open_ports.py", line 8, in get_open_ports
result = s.connect_ex((host, port)) # returns 0 if the connection was successful
^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Scripts\interpreter.exe_main.py", line 7, in
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\core\core.py", line 26, in cli
cli(self)
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\cli\cli.py", line 254, in cli
interpreter.chat()
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\core\core.py", line 88, in chat
for _ in self._streaming_chat(message=message, display=display):
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\core\core.py", line 102, in _streaming_chat
self._llm = setup_llm(self)
^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_llm.py", line 20, in setup_llm
text_llm = setup_text_llm(interpreter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\NAME\AppData\Local\Programs\Python\Python311\Lib\site-packages\interpreter\llm\setup_text_llm.py", line 56, in setup_text_llm
raise Exception("Architecture not yet supported for local LLM inference via Oobabooga. Please run interpreter to connect to a cloud model.")
Exception: Architecture not yet supported for local LLM inference via Oobabooga. Please run interpreter to connect to a cloud model.

grabani · 2023-10-14T10:55:39Z

I am getting the same error.

I am running on:
Windows 11
i5 12th Gen
16GB RAM

ZackBradshaw · 2023-10-14T11:55:05Z

I'm also seeing this error

Traceback (most recent call last):
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/queueing.py", line 406, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1554, in process_api
    result = await self.call_function(
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1206, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/utils.py", line 517, in async_iteration
    return await iterator.__anext__()
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/utils.py", line 510, in __anext__
    return await anyio.to_thread.run_sync(
  File "/home/zack/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/zack/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/zack/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/utils.py", line 493, in run_sync_iterator_async
    return next(iterator)
  File "/home/zack/.local/lib/python3.10/site-packages/gradio/utils.py", line 647, in gen_wrapper
    yield from f(*args, **kwargs)
  File "/home/zack/code/Open-Sourcerer/jarvis.py", line 84, in bot
    for chunk in interpreter.chat(user_message, stream=True, display=False):
  File "/home/zack/.local/lib/python3.10/site-packages/interpreter/core/core.py", line 102, in _streaming_chat
    self._llm = setup_llm(self)
  File "/home/zack/.local/lib/python3.10/site-packages/interpreter/llm/setup_llm.py", line 20, in setup_llm
    text_llm = setup_text_llm(interpreter)
  File "/home/zack/.local/lib/python3.10/site-packages/interpreter/llm/setup_text_llm.py", line 56, in setup_text_llm
    raise Exception("Architecture not yet supported for local LLM inference via `Oobabooga`. Please run `interpreter` to connect to a cloud model.")
Exception: Architecture not yet supported for local LLM inference via `Oobabooga`. Please run `interpreter` to connect to a cloud model.

hyyh619 · 2023-10-14T14:39:37Z

I also have the same bug with windows11, python 3.11.

RedemptionC · 2023-10-15T06:15:05Z

using M1 Pro MacBook, had this issue

gregoryca · 2023-10-15T15:30:15Z

Same here ! strange thing is, i run it from a venv. On a pure Linux (22.04) install it works flawless (albeit slow because the server runs on old hardware). I created the same setup from within WSL2 (Ubuntu 22.04) and now receive the message: "Architecture not yet supported for local LLM inference via Oobabooga."

Sunwood-ai-labs · 2023-10-18T10:34:06Z

I also have the same bug with windows11, Python 3.10.12 on Docker.

reeyarn · 2023-10-18T19:51:50Z

Same here. I run it on Windows 10 + WSL; and again with native Ubuntu. Got the same exception.

frsykml · 2023-10-19T03:41:22Z

Same here, is there a solution?:(

kigenstore · 2023-10-19T03:45:47Z

Same issue, Windows 11 python 3.11

chirayu-2001 · 2023-10-19T12:19:04Z

Same issue, Macbook M1 pro, python 3.11

gerwinvis · 2023-10-19T12:36:08Z

is there an issue with cpu vs gpu?

Sunwood-ai-labs · 2023-10-19T13:34:10Z

It's not the correct solution, but it worked after updating to 0.1.4

pip install open-interpreter==0.1.4

ursgehrig · 2023-10-19T15:24:07Z

Had the same issue with waiting at "Getting started..."; following the instructions of @Sunwood-ai-labs got the interpreter up and running locally again:

C:\Users\u>interpreter --local
A new version is available. Please run 'pip install --upgrade open-interpreter'.

Open Interpreter will use Code Llama for local execution. Use your arrow keys to set up the model.

[?] Parameter count (smaller is faster, larger is more capable): 7B

7B
13B
34B

[?] Quality (smaller is faster, larger is more capable): Small | Size: 2.6 GB, Estimated RAM usage: 5.1 GB

Small | Size: 2.6 GB, Estimated RAM usage: 5.1 GB
Medium | Size: 3.8 GB, Estimated RAM usage: 6.3 GB
Large | Size: 6.7 GB, Estimated RAM usage: 9.2 GB
See More

[?] Use GPU? (Large models might crash on GPU, but will run more quickly) (Y/n): n

Model found at C:\Users\u\AppData\Local\Open Interpreter\Open Interpreter\models\codellama-7b-instruct.Q2_K.gguf

▌ Model set to TheBloke/CodeLlama-7B-Instruct-GGUF

Open Interpreter will require approval before running code. Use interpreter -y to bypass this.

Press CTRL-C to exit.

Guten Tag.

What would you like to do today?

Notnaton · 2023-10-24T16:44:11Z

@ericrallen I cant request review in Ooba, I believe those pull requests will fix many of the: "Getting Started", "Server took too long" and "System not supported" issues on windows

Usuario800 · 2023-11-02T00:16:52Z

Same error

mweichert · 2023-11-04T00:50:23Z

I'm seeing this too. I know that I can run the model locally with ollama, so I find this odd.

Notnaton · 2023-11-04T09:12:54Z

I have a fork of Ooba that fixes this:
https://github.com/Notnaton/ooba

you can install it by running:
pip install git+https://github.com/Notnaton/ooba.git --force-reinstall

mweichert · 2023-11-04T16:35:22Z

Thanks. I installed via the above command then launched open-interpreter again but getting the same error.

…

On Sat, Nov 4, 2023 at 5:13 AM Anton ***@***.***> wrote: I have a fork of Ooba that fixes this: https://github.com/Notnaton/ooba you can install it by running: pip install git+https://github.com/Notnaton/ooba.git --force-reinstall — Reply to this email directly, view it on GitHub <#636 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC3LCZIHNDTDEQQIJFARZTYCYBKFAVCNFSM6AAAAAA56O23TOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJTGM4TCMJYGY> . You are receiving this because you commented.Message ID: ***@***.***>

ericrallen · 2023-11-16T02:43:24Z

I think we can close this one now that LM Studio is powering Open Interpreter’s local models.

Please reopen it or ping me if I’m wrong.

mweichert · 2023-11-19T00:07:07Z

LM Studio isn't available on Linux, so many of us are using Ollama instead.

…

On Wed, Nov 15, 2023 at 9:43 PM Eric Allen ***@***.***> wrote: I *think* we can close this one now that LM Studio is powering Open Interpreter’s local models. Please reopen it or ping me if I’m wrong. — Reply to this email directly, view it on GitHub <#636 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC3LC4ZEQO7YGQQR455QHLYEV4VPAVCNFSM6AAAAAA56O23TOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTG4YDEOJUHE> . You are receiving this because you commented.Message ID: ***@***.***>

ericrallen · 2023-11-19T00:09:44Z

Ollama is great!

I think there’s also a Linux beta of LM Studio, too.

ashakiro added the Bug Something isn't working label Oct 13, 2023

Notnaton mentioned this issue Oct 14, 2023

Some fixes on paths on windows and finding open sockets using async KillianLucas/ooba#4

Closed

mrvic mentioned this issue Oct 22, 2023

MacOS 11.2.3 "not built the support for your system" #667

Closed

This was referenced Oct 24, 2023

Small fixes for windows KillianLucas/ooba#6

Closed

Don't need to scan 8000 ports KillianLucas/ooba#7

Merged

sebvannistel mentioned this issue Oct 26, 2023

Getting stuck at "Setting up the language model..." / Windows / local model #688

Closed

Notnaton mentioned this issue Oct 27, 2023

Getting stuck at "Getting started..." when running local model on Windows PC #665

Closed

ericrallen closed this as completed Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Architecture not yet supported for local LLM inference via `Oobabooga` #636

Architecture not yet supported for local LLM inference via `Oobabooga` #636

ashakiro commented Oct 13, 2023

WonkySoft commented Oct 13, 2023

Kwalladee commented Oct 14, 2023 •

edited

Loading

sebvannistel commented Oct 14, 2023

Notnaton commented Oct 14, 2023 •

edited

Loading

grabani commented Oct 14, 2023

ZackBradshaw commented Oct 14, 2023

hyyh619 commented Oct 14, 2023

RedemptionC commented Oct 15, 2023

gregoryca commented Oct 15, 2023

Sunwood-ai-labs commented Oct 18, 2023

reeyarn commented Oct 18, 2023

frsykml commented Oct 19, 2023

kigenstore commented Oct 19, 2023

chirayu-2001 commented Oct 19, 2023

gerwinvis commented Oct 19, 2023

Sunwood-ai-labs commented Oct 19, 2023

ursgehrig commented Oct 19, 2023

Notnaton commented Oct 24, 2023

Usuario800 commented Nov 2, 2023

mweichert commented Nov 4, 2023

Notnaton commented Nov 4, 2023

mweichert commented Nov 4, 2023 via email

ericrallen commented Nov 16, 2023

mweichert commented Nov 19, 2023 via email

ericrallen commented Nov 19, 2023

Architecture not yet supported for local LLM inference via Oobabooga #636

Architecture not yet supported for local LLM inference via Oobabooga #636

Comments

ashakiro commented Oct 13, 2023

Describe the bug

Reproduce

Expected behavior

Screenshots

Open Interpreter version

Python version

Operating System name and version

Additional context

WonkySoft commented Oct 13, 2023

Kwalladee commented Oct 14, 2023 • edited Loading

sebvannistel commented Oct 14, 2023

Notnaton commented Oct 14, 2023 • edited Loading

grabani commented Oct 14, 2023

ZackBradshaw commented Oct 14, 2023

hyyh619 commented Oct 14, 2023

RedemptionC commented Oct 15, 2023

gregoryca commented Oct 15, 2023

Sunwood-ai-labs commented Oct 18, 2023

reeyarn commented Oct 18, 2023

frsykml commented Oct 19, 2023

kigenstore commented Oct 19, 2023

chirayu-2001 commented Oct 19, 2023

gerwinvis commented Oct 19, 2023

Sunwood-ai-labs commented Oct 19, 2023

ursgehrig commented Oct 19, 2023

Notnaton commented Oct 24, 2023

Usuario800 commented Nov 2, 2023

mweichert commented Nov 4, 2023

Notnaton commented Nov 4, 2023

mweichert commented Nov 4, 2023 via email

ericrallen commented Nov 16, 2023

mweichert commented Nov 19, 2023 via email

ericrallen commented Nov 19, 2023

Architecture not yet supported for local LLM inference via `Oobabooga` #636

Architecture not yet supported for local LLM inference via `Oobabooga` #636

Kwalladee commented Oct 14, 2023 •

edited

Loading

Notnaton commented Oct 14, 2023 •

edited

Loading