Thanks for the review @hlky! For this PR, are there any meaningful tests you would like for me to add? I took a look at FluxIPAdapterTesterMixin, and it's not using MultiIPAdapterImageProjection, so tests for multi ip-adapter would probably need some refactor. Is it relevant atm?

FluxIPAdapterTesterMixin can be updated for multi IPAdapters with IPAdapterTesterMixin as reference.

FluxIPAdapterTesterMixin can be updated for multi IPAdapters with IPAdapterTesterMixin as reference.

Perfect, will do!

Another thing, in #10775 (comment), when you mentioned multi image support for flux-ip-adapter-v2 (to be done on a separate PR), you said ip_adapter_image would need to support List[PipelineImageInput]. You meant the IP-Adapter interface, not the pipeline interface, right? Because atm ip_adapter_image on FluxPipeline is still Optional[PipelineImageInput], and I think we need to update to Optional[Union[PipelineImageInput, List[PipelineImageInput]] (one image per IP-Adapter).

PipelineImageInput is

PipelineImageInput = Union[
    PIL.Image.Image,
    np.ndarray,
    torch.Tensor,
    List[PIL.Image.Image],
    List[np.ndarray],
    List[torch.Tensor],
]

The List is for one image per IPAdapter or one image per ControlNet.

List[PipelineImageInput] would be like in ControlNetUnion where each ControlNet can take multiple inputs (with the experimental scale type, not MultiControlNet).

Ready for review :) I added the ability to mix and match per-layer scale and single scale. For example, you can pass scale for two IP-Adapters as [list_with_19_values, 0.3]. You can also pass a single value and it's used for every layer of every IP-Adapter. I also made use of _is_valid_type to check types as part of processing scale input, so it doesn't fail somewhere later, with minimal additional complexity.

@hlky failed check_code_quality comes from docs/source/en/_toctree.yml

@guiyrt It looks like this has introduced a circular dependency. Let's move _get_detailed_type and _is_valid_type under diffusers.utils and create typing_utils.py cc @yiyixuxu

from ..pipelines.pipeline_loading_utils import _get_detailed_type, _is_valid_type

Just sharing some cool outputs I got, creating image variations by pushing noise through the IP-Adapter using random image embeds instead of outputs from CLIP vision model. Somewhat by accident while testing for ip_adapter_image_embeds, but I really liked the vibe from lucky seed 423 🤩

import torch
from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe: FluxPipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)

pipe.enable_sequential_cpu_offload()

image = pipe(
    width=1024,
    height=1024,
    prompt="A vintage picture of an astronaut in a starry sky",
    generator=torch.manual_seed(423),
).images[0]

image.save('result_no_ipa.jpg')

import torch
from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe: FluxPipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)

pipe.load_ip_adapter(
    "XLabs-AI/flux-ip-adapter",
    weight_name="ip_adapter.safetensors",
    image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14"
)

pipe.set_ip_adapter_scale(0.5)
pipe.enable_sequential_cpu_offload()

image = pipe(
    width=1024,
    height=1024,
    prompt="A vintage picture of an astronaut in a starry sky",
    generator=torch.manual_seed(423),
    ip_adapter_image_embeds=[torch.rand(1, 1, 768)]
).images[0]

image.save('result_single.jpg')

import torch
from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe: FluxPipeline = FluxPipeline.from_pretrained(
   "black-forest-labs/FLUX.1-dev",
   torch_dtype=torch.bfloat16,
)

pipe.load_ip_adapter(
   ["XLabs-AI/flux-ip-adapter", "XLabs-AI/flux-ip-adapter"],
   weight_name=["ip_adapter.safetensors", "ip_adapter.safetensors"],
   image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14"
)

pipe.set_ip_adapter_scale([0.25, 0.25])
pipe.enable_sequential_cpu_offload()

image = pipe(
   width=1024,
   height=1024,
   prompt="A vintage picture of an astronaut in a starry sky",
   generator=torch.manual_seed(423),
   ip_adapter_image_embeds=[torch.rand(1,1,768), torch.rand(1,1,768)]
).images[0]

image.save('result_multi.jpg')

No IP-Adapter	Single IP-Adapter, 0.5 scale	Dual IP-Adapter, 2x 0.25 scale

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.