controlnet arguments unit.image

[dermesut]

this is the arguments that i provide:
controlnet_args = {..., "image": base64_image, "mask_image": base64_mask_image, ...}

this is the output in the terminal:

in get_input_data
        elif (unit_image < 5).all() and (unit_image_fg > 5).any():
    TypeError: '<' not supported between instances of 'dict' and 'int'

if i change the code in "...\extensions-builtin\sd_forge_controlnet\scripts\sdfcontrolnet.py" to:

            unit_image = unit.image
            unit_image_fg = unit.image_fg[:, :, 3] if unit.image_fg is not None else None

            debug = True
            if debug:
                unit_image = unit.image["image"]

            if unit.use_preview_as_input and unit.generated_image is not None:
                image = unit.generated_image
            elif unit.image is None:
                resize_mode = external_code.resize_mode_from_value(p.resize_mode)
                image = HWC3(np.asarray(a1111_i2i_image))
                using_a1111_data = True
            elif (unit_image < 5).all() and (unit_image_fg > 5).any():
                image = unit_image_fg
            else:
                image = unit_image

and
unit_mask_image = unit.image["mask"] a bit further down the road, the terminal doesn't show the error anymore and controlnet works as expected.

this is happening only with the forge-versions that came after the "old" version.

i'm not understanding, what i am expected to provide in my dict - i tried a few variations on the theme, without success.
i've also looked at the output of the extension "api payload", but my payload doesn't seem to differ in a meaningful way.

while this addition to the forge-code might be my workaround, i really would like to not change parts of the forge-code at all.

could someone help me figure it out?

[PopMaRock]

Same issue. ControlNet API call works in A1111 but fails in Forge. I found Reference does not work by api call but IP-Adapter does (even through it still throws a missing parameter error and the type error).

[IgorAherne]

I used the sd-webui-api-payload-display extension, and the payload in browser appears the same as mine in code. But the api won't accept it, unless I generate through browser.

[Jonseed]

I get this error when using controlnet in adetailer through the forge UI. Not sure why this issue was closed as completed. What is the solution?

If I do a "passthrough" in adetailer to the main controlnet integrated, then it works fine. So there seems to be a problem with using controlnet within adetailer?

[samuelm2]

@Jonseed, I had the same issue with using controlnet with adetailer, which I fixed using the suggested code change by @dermesut above. Here is the code change I made (pretty much, switching unit.image to unit.image["image"] and unit.mask_image to unit.image["mask"] in extensions-builtin/sd_forge_controlnet/scripts/controlnet.py.

@@ -197,7 +197,7 @@ class ControlNetForForgeOfficial(scripts.Script):
 
             using_a1111_data = False
 
-            unit_image = unit.image
+            unit_image = unit.image["image"]
             unit_image_fg = unit.image_fg[:, :, 3] if unit.image_fg is not None else None
 
             if unit.use_preview_as_input and unit.generated_image is not None:
@@ -216,7 +216,7 @@ class ControlNetForForgeOfficial(scripts.Script):
 
             image = HWC3(image)
 
-            unit_mask_image = unit.mask_image
+            unit_mask_image = unit.image["mask"]
             unit_mask_image_fg = unit.mask_image_fg[:, :, 3] if unit.mask_image_fg is not None else None
 
             if using_a1111_data:

@lllyasviel is this a safe change to check in? Or is there a better solution?