[Issue]: LCM SDXL LoRa images don't properly render

[ghost]

Issue Description

CFG is set to 1. Batch size 4, LCM sampler selected, 8 steps. SDXL Base 1.0 as model in diffusers, and LoRa loaded as lora:LCM_sdxl:1.0. All extensions disabled

The images look like this:

Is the problem that I renamed the original LoRa so the ui can not detect it? What could be the problem? There is no error message. I know that the quality might suffer a bit because it is LCM but that looks way too bad

Version Platform Description

Version: 044c314
Win 10
RTX 4090
Chrome

Relevant log output

Using VENV: P:\vladnewnew\sdnext-dev\venv
13:23:17-200065 INFO     Starting SD.Next
13:23:17-204066 INFO     Python 3.10.6 on Windows
13:23:17-340064 INFO     Version: app=sd.next updated=2023-11-09 hash=044c3147
                         url=https://github.com/vladmandic/automatic/tree/master
13:23:17-513153 INFO     Platform: arch=AMD64 cpu=AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD system=Windows
                         release=Windows-10-10.0.19045-SP0 python=3.10.6
13:23:17-521234 INFO     nVidia CUDA toolkit detected: nvidia-smi present

Backend

Diffusers

Model

SD-XL

Acknowledgements

• I have read the above and searched for existing issues
• I confirm that this is classified correctly and its not an extension issue

[vladmandic]

run webui --debug
and show log file so i can see that both model and lora sucesfully loaded and lcm sampler was correctly initialized.

[ghost]

14:59:01-101711 INFO     Startup time: 24.01 { torch=5.89 gradio=0.61 libraries=1.45 extensions=4.49 face-restore=0.21 upscalers=0.15
                         ui-extra-networks=0.27 ui-txt2img=0.05 ui-img2img=0.08 ui-settings=0.23 ui-extensions=2.07 ui-defaults=0.07
                         launch=0.23 api=0.09 app-started=0.57 checkpoint=7.40 }
14:59:28-137027 DEBUG    txt2img: id_task=task(bm5v09jdmda9t3t)|prompt=frog
                         <lora:LCM_sdxl:1.0>|negative_prompt=blur|prompt_styles=[]|steps=8|sampler_index=15|latent_index=0|full_quality=
                         True|restore_faces=False|tiling=False|n_iter=1|batch_size=4|cfg_scale=1|clip_skip=1|seed=-1.0|subseed=-1.0|subs
                         eed_strength=0|seed_resize_from_h=0|seed_resize_from_w=0||height=1024|width=1024|enable_hr=False|denoising_stre
                         ngth=0.35|hr_scale=1.4|hr_upscaler=ESRGAN 4x
                         Ultrasharp|hr_force=False|hr_second_pass_steps=28|hr_resize_x=0|hr_resize_y=0|image_cfg_scale=6|diffusers_guida
                         nce_rescale=0.7|refiner_steps=10|refiner_start=0.8|refiner_prompt=|refiner_negative=|override_settings_texts=[]
Loading weights: models\Lora\LCM_sdxl.safetensors ---------------------------------------- 0.0/393.9 MB -:--:--
14:59:29-953053 INFO     Applying LoRA: ['LCM_sdxl'] patch=0.00 load=1.72
14:59:29-957054 DEBUG    Sampler: name=LCM type=all config={'num_train_timesteps': 1000, 'beta_start': 0.0001, 'beta_end': 0.02,
                         'beta_schedule': 'linear', 'prediction_type': 'epsilon'}
14:59:29-960052 DEBUG    Sampler: name=LCM type=scheduler config={'num_train_timesteps': 1000, 'beta_start': 0.00085, 'beta_end': 0.012,
                         'beta_schedule': 'scaled_linear', 'prediction_type': 'epsilon', 'set_alpha_to_one': True,
                         'rescale_betas_zero_snr': False}
14:59:29-966054 DEBUG    Sampler: name=LCM type=model config={'num_train_timesteps': 1000, 'beta_start': 0.00085, 'beta_end': 0.012,
                         'beta_schedule': 'scaled_linear', 'prediction_type': 'epsilon', 'set_alpha_to_one': False,
                         'rescale_betas_zero_snr': False}
14:59:29-970055 DEBUG    Sampler: name=LCM type=user config={'num_train_timesteps': 1000, 'beta_start': 0.00085, 'beta_end': 0.012,
                         'beta_schedule': 'scaled_linear', 'prediction_type': 'epsilon', 'set_alpha_to_one': False,
                         'rescale_betas_zero_snr': False}
14:59:29-977055 DEBUG    Sampler: sampler="LCM" config={'num_train_timesteps': 1000, 'beta_start': 0.00085, 'beta_end': 0.012,
                         'beta_schedule': 'scaled_linear', 'prediction_type': 'epsilon', 'set_alpha_to_one': False,
                         'rescale_betas_zero_snr': False}
14:59:32-097767 DEBUG    Diffuser pipeline: StableDiffusionXLPipeline task=DiffusersTaskType.TEXT_2_IMAGE set={'prompt_embeds':
                         torch.Size([4, 77, 2048]), 'pooled_prompt_embeds': torch.Size([4, 1280]), 'negative_prompt_embeds':
                         torch.Size([4, 77, 2048]), 'negative_pooled_prompt_embeds': torch.Size([4, 1280]), 'guidance_scale': 1,
                         'generator': device(type='cuda'), 'output_type': 'latent', 'num_inference_steps': 8, 'eta': 1.0,
                         'guidance_rescale': 0.7, 'denoising_end': None, 'height': 1024, 'width': 1024, 'parser': 'Full parser'}
Progress  1.37it/s ███████████████████████████████████ 100% 8/8 00:05 00:00 Base
14:59:38-304327 DEBUG    Moving to CPU: model=UNet
14:59:44-285263 DEBUG    VAE decode: name=sdxl_vae (2).safetensors dtype=torch.float32 upcast=True images=4 latents=torch.Size([4, 4,
                         128, 128]) time=5.98
14:59:44-327263 DEBUG    Saving: image="P:\vladnewnew\sdnext-dev\outputs\text\00020--.png" type=PNG size=1024x1024
14:59:44-737221 DEBUG    Saving: image="P:\vladnewnew\sdnext-dev\outputs\text\00021--.png" type=PNG size=1024x1024
14:59:44-941621 DEBUG    Saving: image="P:\vladnewnew\sdnext-dev\outputs\text\00022--.png" type=PNG size=1024x1024
14:59:45-238693 DEBUG    Saving: image="P:\vladnewnew\sdnext-dev\outputs\text\00023--.png" type=PNG size=1024x1024
14:59:45-387738 INFO     Processed: images=4 time=17.17 its=1.86 memory={'ram': {'used': 12.42, 'total': 31.94}, 'gpu': {'used': 8.69,
                         'total': 23.99}, 'retries': 0, 'oom': 0}

If you need startup debug log let me know

[vladmandic]

log looks ok, really not sure why your quality is lower than expected. but given that there are no errors, not much i can go on with.
maybe someone else can chime in.

[ghost]

Given that I already have that problem of chaiNNer not working, I suspect something installed / running in my system is corrupting whatever I install (pythonwise)

What can I do to ... reinstall python? I suspect it does not matter because of the venv, but I am really frustrated with all these problems unique to my system...

[Disty0]

Increase the lora weight to 1.8 or to somewhere around 1.5-2.0

[ghost]

That is interesting, it makes the generations way better, but still with some "pixel grid" in the details. Could go away with a few refiner steps.

https://abload.de/img/00029--bof5g.png
https://abload.de/img/00030--dtex7.png
https://abload.de/img/00031--mviir.png

Almost usable, I do not know if this is the quality to expect (without refiner)?

[bghira]

that looks like mangled pooled embeds.

[eadnams22]

yeah, even at 1.0 strength these results aren’t any better than other samplers at the same steps, and often worse… weird.

[bghira]

damian0815/compel#45 (comment)

this was caused by mishandled text embeds.

[ghost]

Ok, one image there looks exactly like the bug I am having

What can I do about it? Try a different parser? I am not that knowledgeable

[bghira]

not sure. it is probably a code issue where the text embeds are handled. you should try disabling Compel if that is possible. if ti' disabled, try enabling it.

[vladmandic]

specific to LCM lora, can you try setting SD_LORA_DIFFUSER=true env variable and starting the server? it will trigger alternative method i've just added for loading loras. its not capable of handling all loras, but i'm curious on what effect will it have on LCM lora specifically.

[bghira]

it'd be faster if you just test the code before you merge it rather than wait for users to submit bug reports after testing it..

[eadnams22]

it'd be faster if you just test the code before you merge it rather than wait for users to submit bug reports after testing it..

This is an edge case though, not everyone is experiencing it, and one person doesn't have access to every platform to test with.

[bghira]

sorry, how is this an edge case? what is the edge case? you identified the issue?

[vladmandic]

@bghira if i could reproduce at will, i wouldn't ask for tests.
and nobody is forcing you to use free and open source software. if you're not willing to help, then at least try not to be demotivating to people that are - such comments want me just ignore the issue and focus on other things.

[eadnams22]

sorry, how is this an edge case? what is the edge case? you identified the issue?

This strange behaviour with the LCM LORA with SDXL, it doesn't affect everyone, it doesnt even effect every model I've used it with. That's why we reported it, and are willing to help test potential solutions, to figure out whats not working, since we don’t know yet what the issue is and there’s a ton of potential variables that could affect it.

This is the process @bghira , this is how it works.

[qiacheng]

@vladmandic I think it's related to the lora loading method with diffusers backend. if we convert the LDM to LCM first using https://github.com/vladmandic/automatic/blob/master/cli/lcm-convert.py

then do the following:

1. place the model in models/diffusers
2. launch sdnext, load the converted lcm checkpoint
3. use LCM sampler
4. change CFG to 1, steps to 4
5. enter prompt, without any Lora
6. Generate image
7. Correct output

Image generated with LDM + LCM Lora + LCM Scheduler ( low quality image output)

Image generated with the fused LDM+LCM Lora model (Converted using the script above)

[vladmandic]

Yes, that's why I've added alternative lora loading method and asked to test it.

[ghost]

specific to LCM lora, can you try setting SD_LORA_DIFFUSER=true env variable and starting the server? it will trigger alternative method i've just added for loading loras. its not capable of handling all loras, but i'm curious on what effect will it have on LCM lora specifically.

Is it this?

If yes I tried that, does not work. If not, where do I put "SD_LORA_DIFFUSER=true"?

[qiacheng]

@vladmandic confirmed that this commit workaround fixes it 882da27

@antibugsprays to get the above workaround, follow these steps

1. open cmd and cd to sdnext folder

windows:

git checkout dev
git reset --hard
git pull
set SD_LORA_DIFFUSERS=TRUE
webui.bat

linux:

git checkout dev
git reset --hard
git pull
export SD_LORA_DIFFUSERS=TRUE
webui.sh

[ghost]

Thanks a lot, so back to dev for now, will report back once I get home

[vladmandic]

solution has been added to dev branch and will be merged to master in the next release (env variable is no longer needed)