[Feature request] Multi GPU data parelleling

[aeon3]

This is the most intuitive and complete webui fork. It would be amazing if this could be implemented here:

NickLucche/stable-diffusion-nvidia-docker#8

Potential do double image output even with the same VRAM is awesome.

[AUTOMATIC1111]

I'll accept PRs for this as long as they don't make too much changes all over the place.

[ghadeb]

Actually @AUTOMATIC1111 , I believe the changes are limited to 5 files, which are encased on a wrapper for multiprocessing in torch, like the scriplet below (from server.py on the link posted by @aeon3 ):

import gradio as gr
import numpy as np
import torch.multiprocessing as mp
from schedulers import schedulers_names

if name == "main":
mp.set_start_method("spawn", force=True)

from main import inference, MP as model_parallel
prompts = []
def dream(
    prompt: str,
    *args
):
    if not len(prompt.strip()):
        return [], prompts
    images = inference(prompt, *args)
    if not len(prompts) or prompt != prompts[-1]:
        prompts.append([prompt])

I'm unfortunately not a specialist in python, else I'd be glad to help, as the requested feature is something I'm also interested in.

Thanks you and all the credited ppl for all the amazing work.

[mchaker]

I'll be willing to test the code if someone writes it 😅

I have a single system with 8 various GPUs (sizes, architectures, etc) installed.

[ghadeb]

A little more insight into this matter - on the original model, there are several options that enabled the multiprocessing. Inserted in sequence:

--strategy=gpu --auto_select_gpus=true --devices=<num_gpu> --num_nodes=<num_gpu>
You can go a bit more ahead and specify cores, shared memory, etc.

I don't know how if the options are being passed through to the backend stabble-diffusion engine, but I believe if there's a chance to do that, we'll have the functionality working.

[ghadeb]

@AUTOMATIC1111 , I imagine you're really busy with all the requests and bugs, but if you have 5 minutes, have a look at this file on Nickluche's project:

https://github.com/NickLucche/stable-diffusion-nvidia-docker/blob/master/parallel.py

He apparently generated an external wrapper to call the application, allowing it to query if there are or not multi-gpus, and in case there are, data parallel comes into play.

[ghzgod]

This would be a game changer!

[mchaker]

Agreed on this being a game-changer. There are currently some issues that I found and Nick is looking into, but after everything's ironed out, yes -- it would be amazing if this repo used NickLucche's code as a downstream consumer or something. :)

[C43H66N12O12S2]

duplicate of #156