Integrate Kandinsky diffusers pipeline

[seruva19]

These wonderful folks have almost finished implementation of K2 in diffusers: https://www.github.com/huggingface/diffusers/pull/3308
That means I don't have to implement this on my own, which is great xD

To add it to app:

1. https://github.com/seruva19/kubin/blob/master/src/models/model_kd2.py has to be modified to support processing of input params and dispatch them to diffusers pipeline instead of "native" Kandinsky2_1 model
2. argument value "diffusers" for CLI argument "pipeline" wll be responsible for choosing diffusers implementation
3. if diffusers implementation turns out to be more effective than native (and I think it will), "--pipeline=diffusers" will be set as default

[seruva19]

They've done it.
https://huggingface.co/docs/diffusers/main/en/api/pipelines/kandinsky
https://huggingface.co/kandinsky-community
After 0.17 release of diffusers I will start integrating it into the app. I expect to benefit at least from memory and speed optimizations

[user425846]

Does this bring kandinsky closer to being compatible with LoRAs? I tried the diffusers lora dreambooth script but unfortunately it tells me that its not available with multi language models. Any idea? neutron_hare ***@***.***> schrieb am Mo. 29. Mai 2023 um 00:56:

…

They've done it. https://huggingface.co/docs/diffusers/main/en/api/pipelines/kandinsky Time to start integrating it into the app. I expect to benefit at least from memory and speed optimizations <https://huggingface.co/docs/diffusers/main/en/optimization/fp16> — Reply to this email directly, view it on GitHub <#44 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX6YB4BJDPOL72XYLT2MVDTXIPJ2ZANCNFSM6AAAAAAX6JBCLI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[shutter-cat]

Let's hope new optimizations make it work on 8 gb cards

[seruva19]

@user425846 I don't know yet. I hope to adapt diffusers implementation of LoRA. But I haven't got much experience with ML.

[adrianpuiu]

@user425846 I don't know yet. I hope to adapt diffusers implementation of LoRA. But I haven't got much experience with ML.

do these guys have the scripts ready for diffusers ?

[seruva19]

@adrianpuiu Yes, but only in preview version of diffusers. Follow the links I posted.

edit: I misunderstood you, thought you were talking about Kandinsky support in diffusers, not LoRA. Regarding LoRA, I believe generic algorithms they proposed for SD (https://huggingface.co/docs/diffusers/v0.13.0/en/training/lora) can be used (with some changes). I will to try to implement it. Later 🙂

[adrianpuiu]

freaking hell they where fast. just finished reading their the t2i examples . I can't test it atm cause i have meetings with US about mid year priorities... freaking corporations ....but its good cause i work from home :))

[adrianpuiu]

well , clearly it's not implemented in the diffusers-0.16.1-py3-none-any.whl . https://github.com/huggingface/diffusers/tree/kandinsky

[user425846]

I used this guide here to get inference working https://ngwaifoong92.medium.com/introduction-to-kandinsky-2-1-7b7a9131e940 adrianpuiu ***@***.***> schrieb am Mo. 29. Mai 2023 um 13:01:

…

well , clearly it's not implemented in the diffusers-0.16.1-py3-none-any.whl . do you have a dev wheel link ? — Reply to this email directly, view it on GitHub <#44 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX6YB4GF4KE3JOGBYN64U6TXIR6ZJANCNFSM6AAAAAAX6JBCLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[adrianpuiu]

i got it going . thank you

[adrianpuiu]

@seruva19 not really good news .... diffusers eat up an (1) extra gigs of vram . buuttt, the good part is it clears the vram after each generation , compared with the older pipeline that keeps the gpu vram busy ... so, the main conclusion is , its better with diffuers .

How fast can you add it to kubin ? can we keep both pipelines and have end users choose between them ?

[seruva19]

It won't replace old pipeline. As I wrote before,

argument value "diffusers" for CLI argument "pipeline" wll be responsible for choosing diffusers implementation

As for adding it to kubin, I haven't decided yet. First I planned to wait until 'stable' release of diffusers (0.17), but maybe I'll do it earlier.

edit: Also, i think it can be added pretty fast, like 1-2 days at max.

[adrianpuiu]

my thinking is that having the diffuers added more devs will jump on the repo and help evolving it faster

[seruva19]

Either that, or someone will make better frontend for Kandinsky, so you guys won't be tied with my awkward GUI :)

[adrianpuiu]

lol , man, what are you talking about :)) the front look is freaking better then a1111 . :D :D

[user425846]

I dont understand it too, the ui is perfectly fine haha adrianpuiu ***@***.***> schrieb am Mo. 29. Mai 2023 um 15:38:

…

lol , man, what are you talking about :)) the front look is freaking better then a1111 . :D :D — Reply to this email directly, view it on GitHub <#44 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX6YB4GRT4WNJ2TLFJP3QHTXISRHHANCNFSM6AAAAAAX6JBCLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[seruva19]

I don't think so) But thanks anyway 😊