How to choose GPU with AllGPT binding.

[ediweber]

Hello.

First things first: Thank you for creating lollms, it is simply spoken wonderful!

Unfortunately I suffer with the AllGPT binding. I cannot load my GGUF models, output is:

Lollms webui version : 6.8
Listing all extensions
Listing all personalities
Listing models
Loading discussion for client mrQYbMGCR0JO3rJSAAAB
Checking for updates from E:\lollms-webui
update availability: True
Listing all extensions

• Retreiving binding settings
  Requested updating of setting model_name to Naberius-7B-GGUF
  Changing model to: Naberius-7B-GGUF
  Loading model:naberius-7b.Q8_0.gguf
  Folder:E:\loldata\models\gguf
  Error allocating memory ErrorOutOfDeviceMemory
  error loading model: Error allocating vulkan memory.
  llama_load_model_from_file: failed to load model
  LLAMA ERROR: failed to load model from E:\\loldata\\models\\gguf\naberius-7b.Q8_0.gguf
  Couldn't load model: [Model not loaded]
  Traceback (most recent call last):
  File "E:\lollms-webui\app.py", line 914, in update_setting
  self.model = self.binding.build_model()
  File "E:\lollms-webui\zoos\bindings_zoo\gpt_4all_init_.py", line 120, in build_model
  self.model.model.set_thread_count(self.binding_config.n_threads)
  File "C:\Users\eduar\anaconda3\envs\lol\lib\site-packages\gpt4all\pyllmodel.py", line 274, in set_thread_count
  raise Exception("Model not loaded")
  Exception: Model not loaded

I suspect that it tries to use my APU (Ryzen 5600) @ 4GB instead of my Nvidia 3060 - 12 GB, is there a simple way to force it to use the Nvidia card.

Will try with smaller models. In fact it would be nice to have a small one running on the APU and a large one on the dedicated GPU card.

[ediweber]

sorry. I accidently posted this in lollms while this is an lollms-webui issue

[ParisNeo]

Well, in the parameters of gpt4all yo ucan select GPU (cuda)