Implementation of CUDA device id selection (--device-id 0/1/2)

[Extraltodeus]

Hello,

I added the possibility to select which GPU to use with CUDA by a commandline argument "--device-id".

Screenshot after starting on the GPU 1:

Screenshot while loading on GPU 0 (then my session crashed because of a lack of RAM) :

On top of allowing to select which GPU to use, it can allow two sessions in parallel if the system has enough RAM (not my case tho 😅)

[dfaker]

Does this have some advantage over the CUDA_VISIBLE_DEVICES environment variable?

[Extraltodeus]

@dfaker
Even if both GPU are visible by torch, only the first GPU is used. The CUDA_VISIBLE_DEVICES environment variable has no effect for running inferences.

If you mean that modifying that variable allows to select the GPU, then it is only available for single instances use (or perhaps by redeclaring it before starting a second session but this would prevent switching models or could be the source of conflicts if for any reason the instance queries the device again).

Therefore making possible to select a second GPU allows to run two instances at once ON TOP of being simply able to select the device in a cleaner fashion.

[dfaker]

The CUDA_VISIBLE_DEVICES environment variable has no effect for running inferences.

This is news to me! But easily running multiple instances on separate devices makes sense, a little more convenient than chaining an export.

[Extraltodeus]

@dfaker Exactly!

[Lukium]

@Extraltodeus This has been really awesome. I'm currently running a server with 11 3090s thanks to this! I did find some small issues though. Some portions of the code, namely Face Restoration (all samplers) and Highres Fix (DDIM and possibly PLMS) seem to ignore --device-id and just use CUDA:0, which throws an error. I added the bug report here #3713. Is this something that you might be able to look at?

[Extraltodeus]

@Extraltodeus This has been really awesome. I'm currently running a server with 11 3090s thanks to this! I did find some small issues though. Some portions of the code, namely Face Restoration (all samplers) and Highres Fix (DDIM and possibly PLMS) seem to ignore --device-id and just use CUDA:0, which throws an error. I added the bug report here #3713. Is this something that you might be able to look at?

I am unfortunately not having access to a capable enough multi-GPU environment (my only acces has 2 GPU but not enough RAM to run two instances).

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:2! (when checking argument for argument weight in method wrapper__cudnn_convolution)

Looks like either a check should be avoided or a device id precised in the function that gets to that.

Right now I could only test blindly however.

[Lukium]

@Extraltodeus This has been really awesome. I'm currently running a server with 11 3090s thanks to this! I did find some small issues though. Some portions of the code, namely Face Restoration (all samplers) and Highres Fix (DDIM and possibly PLMS) seem to ignore --device-id and just use CUDA:0, which throws an error. I added the bug report here #3713. Is this something that you might be able to look at?

I am unfortunately not having access to a capable enough multi-GPU environment (my only acces has 2 GPU but not enough RAM to run two instances).

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:2! (when checking argument for argument weight in method wrapper__cudnn_convolution)

Looks like either a check should be avoided or a device id precised in the function that gets to that.

Right now I could only test blindly however.

@Extraltodeus sorry I just saw this. I would welcome you into our Discord and would happily test any adjustments to the code in real-time in my environment until we get it figured out. I'm not sure if it's cool to post discord links here but it's in my profile. There's also plenty of other people there (around 400) that can help test.

Thanks again for the awesome addition to the UI.

EDIT: If you do join the Discord, please DM me @Lukium # 0001 and I'll set you up with a @Developer role so you can access all the instances as well.

[zeigerpuppy]

Hi @Lukium I guess you're distributing requests across the multiple web-ui instances using a web proxy.
Do you have any further details of the implementation? I guess HAProxy would work with multiple back-ends and load-balancing mode, but curious what solution you chose!

[Kotori05]

Perhaps the idea of tiled diffsion can be used for multi-GPU support?