-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SDXL with non-truncated prompts results in error #45
Comments
… by truncating prompts
what is the exact error? i see you're doing a batch of 4, do you have the same problem with only one image in a batch? |
the issue occurs regardless of batch size |
riiiight i see, the pooled embeddings have to be truncated. i have pushed an exploratory fix to pypi, can you please try |
|
not sure if the same issue causes this, but i realised after looking into the source that Compel needs |
does this happen if you do a batch size of 1 instead of 4? |
i.e. instead of
just do |
yes, my last tests were all with batch size 1 after your request to do so |
ah, looks like if instead of |
@damian0815 can you try using the first pooled embed vector rather than truncate them? ComfyUI does not truncate. |
sorry i don't know what you mean. the CLIP model only outputs 77 token embeddings. to build a longer embedding you have to actually subdivide the prompt into 75 token chunks and push each chunk through the text encoder separately. in other words truncation isn't something i have to opt in to, it's something that i have to do a lot of engineering to work around. |
well, honestly i haven't dealt with this code successfully in ways that you have. so your understanding is above and beyond mine. but my understanding is that the pooled embed somehow ends up at 104 tokens long here, in these earlier tests. and i'm not sure why that is, but i assumed it was some concatenation going on. |
@bghira i just pushed 2.0.1 with a fix for this issue, you should now be able to run please lmk if you're still having problems, i almost melted my 16GB m1 mac loading the SDXL weights to test out text encoder, was too afraid to try actually generating anything. |
checking |
from compel import Compel, ReturnedEmbeddingsType
from diffusers import DiffusionPipeline
import torch
torch_seed = 123123123
torch.manual_seed(torch_seed)
# SDXL Base with DDIM scheduler, `trailing` timestep spacing.
pipeline = DiffusionPipeline.from_pretrained("ptx0/sdxl-base", use_safetensors=True, torch_dtype=torch.float16).to("cuda")
compel = Compel(
truncate_long_prompts=True,
tokenizer=[
pipeline.tokenizer,
pipeline.tokenizer_2
],
text_encoder=[
pipeline.text_encoder,
pipeline.text_encoder_2
],
returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED,
requires_pooled=[
False,
True
]
)
prompt = 'a cat playing in the forest'
#prompt = "('testing', 'this').and()"
negative_prompt = "a test negative prompt that would be very long indeed" # and maybe we can blow past the seventy seven token limit and like, then we will see the error maybe? it is hard to say because the thing is hard to reproduce here in the standalone script, with the washed out and other studf that usuallyug esosouhfsldfh sldkf aldksfj glasdkjfg lasdkjfg laskdjfgh alsdkfg laskdhfjg alsdfg "
conditioning, pooled = compel(prompt)
negative_embed, negative_pooled = compel(negative_prompt)
[conditioning, negative_embed] = compel.pad_conditioning_tensors_to_same_length([conditioning, negative_embed])
base_strength = 0.50
num_inference_steps = 20
num_images_per_prompt = 1
# generate image
normal_images = pipeline(output_type='pil', num_inference_steps=num_inference_steps, num_images_per_prompt=num_images_per_prompt, width=1152, height=768,
prompt_embeds=conditioning, pooled_prompt_embeds=pooled, negative_prompt_embeds=negative_embed, negative_pooled_prompt_embeds=negative_pooled).images
normal_images[0].save('/notebooks/test0-base.png', format='PNG') |
it's hard to tell but with compel 2.0.1 the long prompt kinda looks a bit distorted and weird, yeah? honestly that's expected. long prompts should't work, they break the math. |
can you post the |
yeah, that's expected because the prompt itself isn't even specifically asking for things that would make sense. for me, the issue is more that the |
Normal short prompt:
and:
|
equates to
brings:
and adding some text to the second segment on the negative, oddly didn't fix it:
|
reducing the positive prompt, down to
|
prompt = "('testing', 'this').and()"
negative_prompt = prompt brings:
|
prompt = "('testing', 'this').and()"
negative_prompt = "('artsy photographs', 'fartsy things no like').and()"
not sure why that one needs to double its size at all when the other doesn't. |
@bghira i made a bunch of fixes to tensor padding and resizing just now - please check out latest |
Hi, I was running into the same issue, and tested yesterday with 2.0.1 and it was fine.
when the negative prompt is longer than the positive prompt:
Now I wanted to fork the repo and try to improve the performance (is it normal that it takes ~500ms?) but it seems that the main branch does not work at all and is back to the old error:
It would be great if you could use GitHub releases or at least tags and push the code before pushing the release to pypi so we can understand the changes of the releases. I am invoking compel like this (basically the same as prompt_embeds, pooled_prompt_embeds = compel.build_conditioning_tensor(prompt)
negative_prompt_embeds, negative_pooled_prompt_embeds = compel.build_conditioning_tensor(negative_prompt)
[prompt_embeds, negative_prompt_embeds] = compel.pad_conditioning_tensors_to_same_length(
conditionings=[prompt_embeds, negative_prompt_embeds]
) |
yeah, sorry about that, i know i've been lazy about it but i've just been putting off getting the Business Processes set up |
will try and make it a priority for v2.0.2 onwards |
fwiw as a stopgap you can use changes to the version string in |
I seem to have a related problem with compel-2.0.1-py3-none-any.whl, this code runs fine:
but as soon as I add .and() syntax changing the prompt like this:
I also tried running the example code https://github.com/damian0815/compel/blob/main/compel-demo-sdxl.py with run_and() resulting in the same error. |
@Sur3 have you tried |
Hi yes thanks updating to compel 2.0.2.dev1 fixes those problems. But I somehow run out of GPU memory when using
instead of
What's the difference between those two syntaxes? |
my abilities start and end at upgrading pip packages ... @damian0815 ? |
good question, idk why you'd be getting a GPU OOM issue. |
please re-open if this is still an error with compel 2.0.2 |
This works:
This does not work:
I'm not sure if there's something I'm doing wrong in this example, so, it's possible that the documentation simply needs an update.
The text was updated successfully, but these errors were encountered: