Help identify actions that cause the VRAM overflow bug

[ostap667inbox]

Problem Description:
After generating the next picture for some reason VRAM is not cleaned. I start generating the next picture and see that the speed has very low and in the resource monitor I see that VRAM is completely full and shared memory is being used.

But I have not yet been able to figure out the exact sequence of actions after which the bug appears. That's why I'm writing here and not in 'Issues'. Maybe you have the same problem and you can help me find a pattern for bug report.

Here's what information I have so far:

• The problem appeared right after the update, after which the 'Never OOM Integrated' tab was added.
• But the 'Enabled for UNet' and 'Enabled for VAE' checkboxes are NOT enabled when the problem occurs.
• The flags: --always-offload-from-vram, --cuda-malloc, --cuda-stream, --pin-shared-memory are NOT specified in the start bat-file
• Pictures are generated without ControlNet. Text2IMG only.
• I use Lightning LoRA (Steps 8, CFG 1, DPM++ 2M SGMUniform).

[ostap667inbox]

Found a pattern. Really annoying bug with VRAM overflow when just adding LoRA to prompt:
#500

[leomaxwell973]

If it helps, I run my box hard as it will let me (least I think i am?), and I've only OOMed since switching to forge when there is a problem with... effectively control-net
(occasionally other modules, but 90% of the time its control-net integrated). I've read of issues between deform and CN as well so make sure you are looking for unofficial forum updates, pretty sure I've patched at least one script file manually, probably 2 (know mov2mov was one without a doubt)).

but yea... I've kinda just avoided integrated CN for now, unless its got something A-detail, or any other extension can't which sadly, is a moderate occurrence, so do get the very occasional OOM, but pretty sure only after touching control-net (and as you described, it don't let go after turning it off, gotta reboot whole thing to swipe your cuda back :(. )

hope my observation/opinion helps :)

Relevant launch commands:

set PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync
set ATTN_PRECISION=fp16
--xformers --api --cuda-stream --pin-shared-memory --all-in-fp16 --cuda-malloc --always-high-vram

P.S.
oh yea, the never OOM extension, seems kinda.... bad... OOMed immediately after turning it on xD but it was probably controlnet at the time again, to be fair, just thought... cmon man... never OOM xD

p.s.s:
I'm aware there is also a launch command to malloc controlnet specifically,, but i forget if that caused launch to crash or not? may see if this whole thing with me and CN is just a me issue or not and it NEEDs to be on the malloc bus with my setup or something, or if forges CN is screwy, guess only time will tell :P