Diffusers IPAdapter bug #839
Comments
strint
added
Request-bug
|
Here is a supported case that you can try using first #837 @joel-simon |
|
I meet a similar problem. ERROR message |
Please try |
|
Thank you, the problem is solved. @ccssu |
|
Hi @ccssu , I tried that before but ran into this error. CodeError
Thank you, |
Your Diffusers version and please try @joel-simon # pipe = compile_pipe(pipe)
pipe.unet = oneflow_compile(pipe.unet) |
|
Thanks, unfortunately that returns the same error. |
|
In case its helpful, I add a print there... |
|
adding these two lines got the unet to compile and full pipeline to compile as well. Almost a fix but then in my app with multiple ipadapters it ignores all but the last one. |
|
get , Please try
|
|
@ccssu ah gotcha, torch >= 2.0 did fix it, thanks! It's not working for multiple ip adapters but I don't think that's supported yet, right? |
|
@ccssu 我有尝试切换到您提供的支持ip adapter 的分支，拉取代码后重新执行你在分支上的demo代码，还是会有这个问题AttributeError: 'list' object has no attribute 'to'，我diffuser是 0.27 torch 是最新的，cuda是12.1。谢谢 |
可以 设置下 |
Please try #837 (comment) , pip install git+https://github.com/huggingface/diffusers.git |
|
@ccssu Thanks, that example does now work! My code with some other modifications does not so I will investigate further and follow up. |
|
@ccssu 您好，打开debug日志后，出现了这样的报错 感谢您的回复 |
|
您好，onediff暂时不支持 python3.11， 请使用 python3.10 。 @cthulhu-tww
|
|
@ccssu okk，我再试试，谢谢您 |
|
@ccssu 您好，按照您所说的，降python版本改为3.10后还是出现这样的报错 |
|
看报错挂在 image_embeds = self.image_encoder(image).image_embeds 应该是用的是 compile_pipe 接口， from onediff.infer_compiler import oneflow_compile
# pipe = compile_pipe(pipe)
pipe.unet = oneflow_compile(pipe.unet)解决2: 可以设置 export VM_REBUILD_DYNAMIC_SHAPE=1 试下。 推荐用 1 diffuser>= 0.27 |
|
@ccssu Hello, I have a minimal script that reproduces the error. It does work for IP adapter; however, it does not support pipe.set_ip_adapter_scale Using torch 2.3.0, python 3.9, diffusers from source 28.0, dev_support_diffusers_ipa branch. It's a different issue, but the script also errors with different image sizes on sd_lightning. `ERROR run got error: <class 'oneflow._oneflow_internal.exception.RuntimeError'> Error: Reshape infered output element count is different with input in op_name: model.down_blocks.1.attentions.0-reshape-484 input shape is : (1,60,60,640) , output shape is : (1,4096,640) , output logical shape is (1,4096,640) , and reshape shape conf is : (1,4096,640) op_loc: ERROR [2024-05-13 21:57:56] /home/ubuntu/onediff/src/onediff/infer_compiler/oneflow/utils.py:23 - Exception in forward: e=RuntimeError('\x1b[1m\x1b[38;2;255;000;000mError\x1b[0m: Reshape infered output element count is different with input in op_name: model.down_blocks.1.attentions.0-reshape-484 input shape is : (1,60,60,640) , output shape is : (1,4096,640) , output logical shape is (1,4096,640) , and reshape shape conf is : (1,4096,640) op_loc: \n') Thanks. |
|
这个分支貌似并不支持动态分辨率，不同的分辨率会触发recompile |
|
无法动态修改ip-adapter的强度，无论如何调整，只会按照首次编译时的scale进行推理 @ccssu |
|
@ccssu hi, any update on the ipadapter weight issue I mentioned above? Thanks, |
|
|
@lqfool thanks, unfortunately that example did not work. On commit f3f7e4e Script works ok without compilation Thanks
|
|
@ccssu any updates here? I need ipadapter support to use this library. Thank you very much. |
|
@joel-simon please add a new issue for this. This one is too old and too long to follow. |
Describe the bug
I have the baseline text_to_image_sdxl_light.py working (using lora checkpoint) and tried adding ipadapter to it. Any ipadapter examples would be great, ideally I'm looking to have multiple ip adapters as well as per https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#multi-ip-adapter.
Thank you very much.
Your environment
OS
ubuntu
version: 0.9.1.dev20240420+cu118
git_commit: 665bcf8
cmake_build_type: Release
rdma: True
mlir: True
enterprise: False
How To Reproduce
Steps to reproduce the behavior(code or script):
Modify text_to_image_sdxl_light.py
add
pipe.load_ip_adapter( "h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter-plus_sdxl_vit-h.safetensors", ) pipe.set_ip_adapter_scale(0.6)then run with
`from diffusers.utils import load_image
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_diner.png")
print("Warmup with running graphs...")
torch.manual_seed(args.seed)
image = pipe(
prompt=args.prompt,
height=args.height,
width=args.width,
num_inference_steps=n_steps,
guidance_scale=0,
ip_adapter_image=image,
output_type=OUTPUT_TYPE,
).images
`
The complete error message
The text was updated successfully, but these errors were encountered: