[Feature Request] Add flags to run various models on the CPU.

[bent95]

I have a GTX-16xx GPU with just 4GB of VRAM, but thanks to all of the optimizations I am still able to generate images at pretty high resolution. However there are other parts of the program that still causes CUDA out of memory errors for me:

I get the error when trying to upscale images with realesrgan that are larger than 512x512. By googling around I figured out how to run it on the CPU instead by adding a couple of lines of code to “upscale_with_realesrgan”. This fixed the problem, and performance is still acceptable. However it would be nice to have this fix implemented as an option or command line argument, as there are probably others having the same problem.

I also get the error when using the CLIP interrogator; the BLIP-caption generates just fine, but I get an out of memory error within the “rank”-function when trying to generate artist descriptions. Here I did not figure out how to run it on the CPU, but if it is doable I figured it might be a quick fix as well?

Lastly I just want to thank you for making this, it has been a real joy to use!

[AUTOMATIC1111]

For interrogator I added code to run it on 4GB. Try it now.
For other issues, they are valid, but I will likely not do them anytime soon. If someone makes a PR that would be welcome.

[bent95]

Unfortunately I still get an out of memory error when running the interrogator. For the record I am using a GTX 1650 super, and running the program with the following command line arguments:

--lowvram --precision full --no-half --always-batch-cond-uncond --unload-gfpgan –opt-split-attention

Here is the full error:

Error interrogating
Traceback (most recent call last):
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\modules\interrogate.py", line 156, in interrogate
matches = self.rank(image_features, items, top_count=topn)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\modules\interrogate.py", line 102, in rank
text_features = self.clip_model.encode_text(text_tokens).type(self.dtype)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\clip\model.py", line 348, in encode_text
x = self.transformer(x)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\clip\model.py", line 203, in forward
return self.resblocks(x)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
input = module(input)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\clip\model.py", line 191, in forward
x = x + self.mlp(self.ln_2(x))
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
input = module(input)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\stable-diffusion\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\clip\model.py", line 168, in forward
return x * torch.sigmoid(1.702 * x)
RuntimeError: CUDA out of memory. Tried to allocate 678.00 MiB (GPU 0; 4.00 GiB total capacity; 2.05 GiB already allocated; 0 bytes free; 3.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[MrCheeze]

I get the same error when using the interrogate function. (GTX 1660 Ti, 6GB VRAM).

[HPPinata]

@bent95 These options now exist (see this page), this should be closed.