ControlNet via API issues

[ratzycon]

Hi 👋😄

Trying to get controlnet from API to work with Forge.

no matter what i try from the API i get the same error and the controlnet image fails to apply, while it works inside the forge editor ui
the same script outputs correctly if run against regular webui

errors are thrown in the console when running it via api (KeyError: 0)

What am I doing wrong? :S or is there a bug?

this is the test script i'm using:

import io
import cv2
import base64
import requests
import json
from PIL import Image, PngImagePlugin

def test(test_image):

    url = "http://localhost:7860" # change to your SD url

    img = cv2.imread(test_image)
    retval, bytes = cv2.imencode('.png', img)
    encoded_image = base64.b64encode(bytes).decode('utf-8')

    # Note: You can no longer pass file path in payload as in sd-webui-controlnet.

    payload = {
        "prompt":"2girls, reading, book, elf, simple white background, white dress, hatsune miku",
        "negative_prompt": "",
        "sampler_name": "Euler a",
        "batch_size": 1,
        "steps": 20,
        "cfg_scale": 6,
        "width": 1024,
        "height": 1024,

        # txt2img should not need any init images except for the controlnet image
        #"init_images": [test_image],
        #"init_images": [encoded_image],
        
        "alwayson_scripts": {
            "controlnet": {
                "args": [
                    {
                        "enabled": True,

                        "image": encoded_image,
                        #"mask_image": encoded_image,

                        "control_mode": "Balanced", 
                        "guidance_end": 1,
                        "guidance_start": 0,
                        "pixel_perfect": True,
                        "processor_res": 512,
                        "resize_mode": "Resize and Fill",  # "Just Resize", "Crop and Resize", "Resize and Fill"
                        "threshold_a": 0.5,
                        "threshold_b": 0.5,
                        "weight": 1,
                        "save_detected_map": True,
                        "hr_option": "Both",

                        #"module": "canny",
                        "module": "lineart_anime",
                        
                        "model": "t2i-adapter_diffusers_xl_canny [6b0c1490]",
                        #"model": "control_v11p_sd15_canny [d14c016b]",
                        #"model": "control_v11p_sd15s2_lineart_anime [3825e83e]",
                    }
                ]
            }
        }
    }

    response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)

    r = response.json()
    result = r['images'][0]
    image = Image.open(io.BytesIO(base64.b64decode(result.split(",", 1)[0])))

    pnginfo = PngImagePlugin.PngInfo()
    if 'parameters' in image.info and image.info['parameters'] is not None:
        pnginfo.add_text("parameters", image.info.get('parameters'))
        print("\nParameters:\n" + json.dumps(image.info, indent=4))

    image.save('test_output.png', pnginfo=pnginfo)

    print("image saved to test_output.png")

test("test.png")

controlnet test image:

controlnet console error:

controlnet webui in editor output with same settings (no console errors):

test output using API using Forge:

test output if run with API against regular webui:

controlnet_test.zip

[dermesut]

i believe this to be the same issue i posted here: #1059 (comment)
the code-change i suggested there (see "debug = True" and the following) makes it work again.
but be aware: that code change makes it NOT work in the webui. so either, you have it working via api or via webui (by undoing the code-change and restarting the server). not both in one server session.

[ratzycon]

couldn't tell if it's the same bug,
it's not the same error i see in console tho

in either case it seems there is a difference between api and the editor
sadly it's not an option in my case to change things so it only runs with api,
i use the editor for testing on the side while discord bots run via api

would maybe be good to have some unit tests that run to check api / editor are consistent in outputs with similar payloads

from your post

unit_image = unit.image

debug = True
if debug:
    unit_image = unit.image["image"]

we are clearly setting the image in the unit args directly to be "image" = image64 in the payload, why would it be stored in unit.image["image]?

[dermesut]

...it's not the same error i see in console tho...
i believe it is though. you show what i believe to be the last part of the error messages. it shows the consequence of a prior error message, which is "elif (unit_image < 5).all() and (unit_image_fg > 5).any():
TypeError: '<' not supported between instances of 'dict' and 'int'" .
if i'm correct, you should be seeing this, too if you scroll up a bit in your terminal.

[ratzycon]

yea looks like you are right,
is there are bug report for this? found other reports but they were closed as someone just said "now its working" but seems it isn't working

[dermesut]

...why would it be stored in unit.image["image]?...
i am not able to answer why-questions, quite franky i have too little experience in those things.
what i can tell you, is what i already told you.
and: in the commit that changed things (e26abf8#diff-f2d2c2c48b8885e00c9e9892978d5cc88ece784483f73ee85a6b2cd19917af9f) - see line 206 for example - one can see, the value of the key "image" was being fetched to to things.
i'm guessing, changes in other parts of the code-base, that respected this change in controlnet.py, have not been done, when it comes to the api-context.
i have not been able to gain an overview of where's what, so... this is the workaround i'm using, to be able to use controlnet within my app at all.
there are other things - of course - that are not working with api. also in the controlnet-context. e.g. there used to be a list of images in the response, containg - among others - the preprocessor results of each controlnet unit.
those are not delievered anymore, as far as i can see.
etc.

[dermesut]

is there are bug report for this?
wel, i understood the "issues"-section as the place to post bugs.
that's what i intended with my posting, i was referring to above (#1059 (comment)).
but it does not seem to get us, what we'd need.

[ratzycon]

ok cool, maybe the issue doesn't sound dramatic enough, like 'Controlnet api not working!' xD
didn't show up when i was searching for controlnet api bugs

thank you for putting a clear light on the issue

i posted here first, because

1. it seems there have been multiple reports that got closed, so wanted to see if i was just messing something up in payload
2. yea it seems there is more responses in discussions,

maybe i'll convert it to an issue, we might need more focus on it to get a fix, altho your issue report show it pretty clear

checking the commit you linked, the changes shows they changed from the .image property to just contain the data, instead of a .image['image'] and .image['mask] - but that is what we are sending in the payload, right? .image = image64,

just to be clear, you mean the api code somewhere is still trying to use the unit_image = unit.image["image"]?
or is it just that it tries to enumerate an int or something since it's not a list

[dermesut]

just to be clear, you mean the api code somewhere is still trying to use the unit_image = unit.image["image"]? or is it just that it tries to enumerate an int or something since it's not a list
i couldn't tell, really. as i mentioned, i have yet to win some kind of an oversight of the logic-flow in terms of the api-context.

asking chatgpt, what (unit_image < 5).all() does and what happens if unit_image is a dict, i get what is shown as an error in the terminal:
https://chatgpt.com/share/b2bb4f00-2e47-4576-9d8f-daeca89dbaad

so i'm guessing, somehow somewhere the payload we're providing via api misses the one or the other step.
while it is treated as a dict via api, in the webui-version of the process it seems to become that boolean array, that chatgpt is mentioning - thus not making any problems in the webui.

but that's guess work. not worth a dime.