[Official callbacks] SDXL Controlnet CFG Cutoff

[asomoza]

What does this PR do?

Adds a new official callback for the SDXL Controlnet pipeline.

After testing with different alternatives, I found that just adding a new callback for this specific pipeline was the best and simpler solution.

Fixes #8686

How to test

import random
from datetime import datetime

import torch

from diffusers import AutoencoderKL, ControlNetModel, StableDiffusionXLControlNetPipeline
from diffusers.callbacks import SDXLControlnetCFGCutoffCallback
from diffusers.utils import load_image


anyline = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/callbacks/20240829020431.png"
)

controlnet = ControlNetModel.from_pretrained("TheMistoAI/MistoLine", torch_dtype=torch.float16, variant="fp16")
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

prompt = "high quality photo of a yellow plastic huggingface emoji toy, white background"
callback = SDXLControlnetCFGCutoffCallback(cutoff_step_ratio=0.3)

image = pipe(
    prompt,
    num_inference_steps=25,
    image=anyline,
    controlnet_conditioning_scale=0.6,
    controlnet_guidance_end=0.9,
    callback_on_step_end=callback,
).images[0]

image.save("cnet_callback.png")

anyine	Normal	Callback

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@rootonchair @juancopi81

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[rootonchair]

LGTM. Just one concern if this issue exists in other callbacks

[asomoza]

yeah, this was discussed in the original PR, we probably need to do this kind of callbacks to all the different pipelines, so this means inpaint, img2img and for different model archs.

But this is the only one so far that someone asked for it, so maybe lets evaluate to do them when someone asks for an official callback, also the community is open to do PRs with them.

Also probably we need to start naming the args more consistently, for this pipeline the controlnet image is image but for the inpaint and img2img pipelines this is for the source image and this one is named control_image.

[juancopi81]

Thanks @asomoza!!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

thanks!

[kunc01]

yeah, this was discussed in the original PR, we probably need to do this kind of callbacks to all the different pipelines, so this means inpaint, img2img and for different model archs.

But this is the only one so far that someone asked for it, so maybe lets evaluate to do them when someone asks for an official callback, also the community is open to do PRs with them.

Also probably we need to start naming the args more consistently, for this pipeline the controlnet image is image but for the inpaint and img2img pipelines this is for the source image and this one is named control_image.

@asomoza Hello, I'm trying SDXL with multicontrolnets, using the code in this PR leads to the same error as #8686. I've tried replace this line with
image = image[-1].chunk(2)
image = [image[0] + image[1]]
but the result is poor with cutoff_step_ratio=0.4. Any help is appreciated!

[rootonchair]

@kunc01 you should open a new issue and provide example code