Running --local load my local model over and over again everytime a message is sent

[xinranli0809]

Describe the bug

I managed to load a local mistral 7b model and was able to chat back and forth with the interpreter, but the time it takes to begin streaming the tokens is extremely long after I send a message, considering I get decent token/s once it starts streaming.

Upon a closer look at the memory usage in the task manager, I see that every time I press enter and send a message, the program loads a brand new Ooba server and a GGUF model! This is confirmed by adding a print("Ooba starting and model loading!") before Line 84 in Ooba's llm.py and that an additional python process will pop up in the task manager and occupy 8GB RAM after sending every message.

Please see the screenshots for details.

Reproduce

1. interpreter --local
2. chat with it and see how long it takes before begin streaming the response
3. Monitor your RAM usage, the model is loaded into RAM every time you send a message

Expected behavior

1. Model load only once per session

Screenshots

Open Interpreter version

0.1.10

Python version

3.11.6

Operating System name and version

Windows 10

Additional context

I am also getting an interpreter.procedure being a NoneType error, for some reason... But I don't think these are related issues. I tried putting the procedures_db.json in the same directory as get_relevant_procedures_string.py and loaded it manually to make sure it was presenting. The NoneType error went away but a new model is still being loaded after every message.

[Notnaton]

There are two bugs here

• [Fixed] Fix a crash in get_relevant_procedures_string #676
• [Waiting] Fixes restart every second message KillianLucas/ooba#8
  Hopefully solved next update

[xinranli0809]

There are two bugs here

• [Fixed] Fix a crash in get_relevant_procedures_string #676
• [Waiting] Fixes restart every second message ooba#8
  Hopefully solved next update

It works, thanks!

[ericrallen]

It works, thanks!

Hey there, @xinranli0809!

Does that mean this issue is resolved?

[Notnaton]

It works, thanks!

Hey there, @xinranli0809!

Does that mean this issue is resolved?

@ericrallen
This is an ooba issue, once ooba is merged with KillianLucas/ooba#8 it should be fixed.

[ericrallen]

Thanks for all of the awesome work, @Notnaton!