[Bug]: hires fix stopped working

[GalaxyTimeMachine]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

Just updated to commit 4dbde22 and not getting good results. Exactly the same prompts, settings and seed are not giving the same as before, and I'm getting a lot of body horrors (double length bodies, 2 pairs of boobs etc).

I think it may be related to the hires fix. I'm making 1024x768 images and hires fix used to get rid of body distortions...it's not any more. No distortions with lower res images.

Steps to reproduce the problem

1. Go to ....
2. Press ....
3. ...

What should have happened?

Same image results as before this commit, when using identical prompts, models, settings and seeds.

Commit where the problem happens

4dbde22

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--no-half-vae --xformers --listen --port 16996 --api --cors-allow-origins=https://www.painthua.com

Additional information, context and logs

No response

[GalaxyTimeMachine]

Same in 8d12a72

[Rogal80]

Hi, same here, and lately, I was using https://github.com/Kahsolt/stable-diffusion-webui-sonar which gives really good results, but after last commit it stopped wok showing this issue :
Traceback (most recent call last):
File "I:\Github\stable-diffusion-webui\modules\call_queue.py", line 45, in f
res = list(func(*args, **kwargs))
File "I:\Github\stable-diffusion-webui\modules\call_queue.py", line 28, in f
res = func(*args, **kwargs)
File "I:\Github\stable-diffusion-webui\modules\txt2img.py", line 46, in txt2img
processed = modules.scripts.scripts_txt2img.run(p, *args)
File "I:\Github\stable-diffusion-webui\modules\scripts.py", line 328, in run
processed = script.run(p, *script_args)
File "I:\Github\stable-diffusion-webui\extensions\stable-diffusion-webui-sonar\scripts\sonar.py", line 951, in run
proc = process_images(p)
File "I:\Github\stable-diffusion-webui\extensions\stable-diffusion-webui-sonar\scripts\sonar.py", line 78, in process_images
res = process_images_inner(p)
File "I:\Github\stable-diffusion-webui\extensions\stable-diffusion-webui-sonar\scripts\sonar.py", line 180, in process_images_inner
samples_ddim = sample_func(p, conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
File "I:\Github\stable-diffusion-webui\extensions\stable-diffusion-webui-sonar\scripts\sonar.py", line 271, in StableDiffusionProcessingTxt2Img_sample
x = create_random_tensors([opt_C, self.firstphase_height // opt_f, self.firstphase_width // opt_f], seeds=seeds, subseeds=subseeds, subseed_strength=self.subseed_strength, seed_resize_from_h=self.seed_resize_from_h, seed_resize_from_w=self.seed_resize_from_w, p=self)
AttributeError: 'StableDiffusionProcessingTxt2Img' object has no attribute 'firstphase_height'

Without sonar

With sonar

[GalaxyTimeMachine]

I think you should go and log that on the other repo. I don't get the errors you mentioned, and different image results to your examples, so I'm not convinced it's the same problem.

This is with commit fd4461d

This is how it looks with the current commit

[Rogal80]

it same problem with highres.fix like you @GalaxyTimeMachine mention but i am also using sonar script with was written for previous highrest.fix - as you seee in console issue right now there is no "firstphase_height" in new highres.fix

[Rogal80]

I think you should go and log that on the other repo. I don't get the errors you mentioned, and different image results to your examples, so I'm not convinced it's the same problem.

This is with commit fd4461d

This is how it looks with the current commit
I have all same issue – previously in low res pass there was a lot deformation and highres pass was nicely fixing those

[GalaxyTimeMachine]

If anyone wants a temporary fix, you can switch to the last working branch using:

git checkout fd4461d44c7256d56889f5b5ed9fb660a859172f

To change back:

git checkout master

[E-16]

I got identical results by using Hires. fix on 8d12a72 and fd4461d with same prompt and settings.
Only thing I change is width and height to make sense "Upscale by 2" (btw, much thanks for this change, now it's way easier to switch)

[GalaxyTimeMachine]

I got identical results by using Hires. fix on 8d12a72 and fd4461d with same prompt and settings. Only thing I change is width and height to make sense "Upscale by 2" (btw, much thanks for this change, now it's way easier to switch)

I tried that, but the images are then completely different to what they used to be. All you're doing is creating a low res image and upscaling it, that's not the same as hires fix.

[Rogal80]

Thx @GalaxyTimeMachine after rollback - higresfix works perfect

[E-16]

I got identical results by using Hires. fix on 8d12a72 and fd4461d with same prompt and settings. Only thing I change is width and height to make sense "Upscale by 2" (btw, much thanks for this change, now it's way easier to switch)

I tried that, but the images are then completely different to what they used to be. All you're doing is creating a low res image and upscaling it, that's not the same as hires fix.

I didnt use img2img if that's your question

I set some prompt, set a constant seed, enabled Hires. fix, generated 1024x1024 in the version fd4461d got the result.
Switched to new version: 8d12a72
Set all parameters same, enabled Hires. fix, due to the fact that there is now not a size target, but a multiplier instead, resized the original image to 512x512, generated it, got exactly the same picture with a size of 1024x1024, as it in the version fd4461d (pixel to pixel, byte to byte)

[GalaxyTimeMachine]

Tried it with "Euler a"? You'll get a different image every time you change the initial size. They won't (shouldn't) be the same. Mine aren't with most, if not all, samplers.

[E-16]

Tried it with "Euler a"? You'll get a different image every time you change the initial size. They won't (shouldn't) be the same. Mine aren't with most, if not all, samplers.

Checked "Euler a", same results

are you sure your Upscaler type in Hires. fix matches as same as in Settings?

[GalaxyTimeMachine]

Tried it with "Euler a"? You'll get a different image every time you change the initial size. They won't (shouldn't) be the same. Mine aren't with most, if not all, samplers.

Checked "Euler a", same results

are you sure your Upscaler type in Hires. fix matches as same as in Settings?

Where in settings? This is for txt2img, not img2img.

[E-16]

Tried it with "Euler a"? You'll get a different image every time you change the initial size. They won't (shouldn't) be the same. Mine aren't with most, if not all, samplers.

Checked "Euler a", same results
are you sure your Upscaler type in Hires. fix matches as same as in Settings?

Where in settings? This is for txt2img, not img2img.

[GalaxyTimeMachine]

Tried it with "Euler a"? You'll get a different image every time you change the initial size. They won't (shouldn't) be the same. Mine aren't with most, if not all, samplers.

Checked "Euler a", same results
are you sure your Upscaler type in Hires. fix matches as same as in Settings?

Where in settings? This is for txt2img, not img2img.

The results are still different versions of a similar image.

[E-16]

if you curious how settings looks like

[E-16]

just checked, non-square pictures are not the case either, still got same pictures on both versions

[Neverdusk]

Could an option be added to use the old system? The new system's odd multipliers seem to give me strange outputs now. I also didn't always upscale width and length by the same amount, but the new system forces the same upscale on both axes. So I can't recreate old images the same way.

[GalaxyTimeMachine]

Settings are same on both versions:

This is fd4461d

This is (just updated again) 251ecee

Clearly 2 different images with the same seed, prompt, model, but different commits from here.

[GalaxyTimeMachine]

If I set "upscale by" to 1 and the size to 768x1024, I get this.

[E-16]

Settings are same on both versions:

This is fd4461d

This is (just updated again) 251ecee

Clearly 2 different images with the same seed, prompt, model, but different commits from here.

I looked at your settings and I already understanding your problem, your initial values of the first pass are 0 and 0, which means default values, which means 512x512 values (to match almost all trained models base resolution), therefore in your case the picture was matched the hires size and then the edges just got cut off, from this you get a different results, in fact your GPU basically doing useless work, because it generates something that is not then used

[GalaxyTimeMachine]

Settings are same on both versions:
This is fd4461d
This is (just updated again) 251ecee
Clearly 2 different images with the same seed, prompt, model, but different commits from here.

I looked at your settings and I already understanding your problem, your initial values of the first pass are 0 and 0, which means default values, which means 512x512 values (to match almost all trained models base resolution), therefore in your case the picture was matched the hires size and then the edges just got cut off, from this you get a different results, in fact your GPU basically doing useless work, because it generates something that is not then used

Interesting, thanks! Would that mean there is no way to replicate the original image with the latest changes? I'd say that still needs fixing.

[GalaxyTimeMachine]

I looked at your settings and I already understanding your problem, your initial values of the first pass are 0 and 0, which means default values, which means 512x512 values (to match almost all trained models base resolution), therefore in your case the picture was matched the hires size and then the edges just got cut off, from this you get a different results, in fact your GPU basically doing useless work, because it generates something that is not then used

That was spot on! I just re-ran the original prompt, in fd4461d, using passes of 384x512 and I now get the same image as I would in the latest master.
It still doesn't give a way to re-create older gens, but does explain why it's happening.

[a-l-e-x-d-s-9]

As far as I understand the old hires-fix always used img2img upscaler from the settings, but it was missing from PNG info. It's good that it is added into PNG info now. Also, it's good that upscaler can be easily changed from hires-fix.
But I don't think that using a multiplier is a good idea, it's confusing, it should be the actual image size. I care about the size of the big image, because I am limited by it. Now I need to set the small resolution, then calculate the multiplier. It's a terrible user experience.
I would prefer the old system + option to lock the ratio between high and low resolution.

[E-16]

As far as I understand the old hires-fix always used img2imd upscaler from the settings, but it was missing from PNG info. It's good that it is added into PNG info now. Also, it's good that upscaler can be easily changed from hires-fix. But I don't think that using a multiplier is a good idea, it's confusing, it should be the actual image size. I care about the size of the big image, because I am limited by it. Now I need to set the small resolution, then calculate the multiplier. It's a terrible user experience. I would prefer the old system + option to lock the ratio between high and low resolution.

If something needs to be changed, then I prefere to make inside Hires. fix width and height as target values (not start values) instead of the multiplier, i.e. swap the values from old ver, ​​so that you can switch just as easily as in new version as well as have full compatibility with the old version

[a-l-e-x-d-s-9]

If something needs to be changed, then I prefere to make inside Hires. fix width and height as target values (not start values) instead of the multiplier, i.e. swap the values from old ver, ​​so that you can switch just as easily as in new version as well as have full compatibility with the old version

Agree, why not save the low and the high resolutions as it was before?! That way all the settings are well defined, without the need to convert values with a multiplier. The multiplier can be saved as an additional value, although it is redundant, unless they can mismatch. And backward compatibility is needed.

[E-16]

What I would want to see is something like this

it becomes a little messy, arent it... But since these settings are inside Highres. fix, therefore they wont hardly affect the interface, I think these settings are more pleasant for people to see

PS: I just realized something, it's no longer a Highres. fix (including last update too), more like HighRes applyer or just HighRes

[manzomium]

git checkout fd4461d44c7256d56889f5b5ed9fb660a859172f

sorry I'm a total newbie to this. Where do I need to type that? Thanks <3

[Flonixcorn]

well, can you not just create the base 512x512 image with the same image prompt? could you try that and see if it generates different images?

[E-16]

@Flonixcorn
You also dont fully understand the point of the problem itself.
I'm hands down for a new features (I even dropped my previous iteration and started almost from scratch new one, because I see a new potential), but you need to understand one thing, this feature has flaws and problems that I pointed out here #6219 (comment)

I dont want to rollback this Hires fix new feature, or make a separate compatibility button, while we can just complete this feature so that everyone will be happy.

Currently this feature has 2 main problems (specifically talking about the multiplier):

For non-square pictures, its impossible to create a square firstpass layer (for which all models was trained)

And this is A BIG DEAL, because the models (checkpoints) are designed to generate firstpass layer as it was conceived by the creator of the model, means SQUARED (In most cases 512x512).

Because of the multiplier step, we get very strange values, for example 512*2.05 = 1049.6

Although rounding up to 8 is implemented there, the values still turn out strange, which cannot be set in the usual step (64x), unless you manually change the step value into ui-config.json (which I definitely don't recommend, unless you want to struggle with black pictures)

The best solution in my eyes is make it like this #6219 (comment) which will keep new feature (firstpass resolution replaced by target resolution, swapped) as well as make everyone happy (with having a backward compatibility)

[Flonixcorn]

Very true, i also liked the old one, and a compability would be nice

[Quark999]

Apparently, the old upscaler was ESGRAN?

[ghost]

My images generated with highres. fix are identical to how they were before and after this rework. I'm not sure how numerous people are running into issues with this.
It makes sense when you compare the actual diff, the code is essentially identical besides reworking it to not crop the final output.

[E-16]

@stysmmaker

My images generated with highres. fix are identical to how they were before and after this rework. I'm not sure how numerous people are running into issues with this. It makes sense when you compare the actual diff, the code is essentially identical besides reworking it to not crop the final output.

I also had no problems with new version, just adapted all settings to the new ones, and successfully continued to generate.
But the lack of cropping (or more like the control of the firstpass layer) became the main problem of this whole issue topic.

So whats the problem?

In simple words, for non-square pictures, for example, people want something like: 512x512 -> 768x1024, but now they dont have the opportunity to make 512x512 -> 768x1024, only 384x512 -> 768x1024, which I think you understand that cause the result different, but it wouldnt be the end of the world, like adapt to a new one, if it werent of HOW DIFFERENT this result is (basically just awful). For them, it doesnt matter that firstpass layer image was cropped (its also generated in seconds, not a big deal), the main thing that they care is that the firstpass layer image is generated correctly.

Why exactly 512x512?

Because models have been trained for this specific values and produce best result at this specific resolution (there are also 768x768 models)

The solution I would like to see #6219 (comment) replace Upscale by slider in two sliders HighRes Width and HighRes Height

but it will not be so easy, because for example parameter like Hires upscale: 2 already exists in new generated pictures

[ghost]

That's a fair compromise I think, even if I do vastly prefer the Upscale by functionality now.

For the record, voldy/AUTOMATIC has stated before he does not bother checking issues/discussions, so if you want his attention someone will have to make a PR.

[E-16]

If I knew python, I would have done PR at this point, but well, we will probably wait for someone who can do it

a hundred comment, god damn

[Edu2014]

technically it was supposed to be identical

[mart-hill]

Apparently, the old upscaler was ESGRAN?

After some testing, I believe the Latent upscaler was the default upscaler for "old" hiresfix.

[brodieferguson]

These are the results I'm getting now using 2x Latent upscale, DPM++ Samplers. Before the results were photoreal. Something is badly broken for me.

[mart-hill]

These are the results I'm getting now using 2x Latent upscale, DPM++ Samplers. Before the results were photoreal. Something is badly broken for me.

Is it possible to test it? I mean, model/embedding/hypernetwork?

[brodieferguson]

Is it possible to test it? I mean, model/embedding/hypernetwork?

Seems to happen to me on any model. That was wavymulder modelshoot, but it happens on base 1.5 for me even. I did a fresh git clone too after wiping everything.

[mart-hill]

Is it possible to test it? I mean, model/embedding/hypernetwork?

Seems to happen to me on any model. That was wavymulder modelshoot, but it happens on base 1.5 for me even. I did a fresh git clone too after wiping everything.

Could you try this interface and test, if it also returns garbled images? I then switched to "beta" internally, in the UI configuration, and re-ran that UI, to install everything (there's more functionality in beta).

[Kahsolt]

Hello everyone 🍭 , we know this is how hires.fix actually works: txt2img -> upscale -> img2img.
Recent updates mainly changed behaviour of the upscale step, keeping other two steps not touched.

For the legacy upscale operation, here's the implementation details:

• (launch txt2img, generate latent images sized firstpass width/height)
• center crop the txt2img samples, truncating size to match final target width/height in aspect ratio
• resize RGB image or latent image to width/height
  • if option Upscale latent space image when doing hires. fix is checked, you have no other choice but to do bilinear interpolation on latent image in torch
  • otherwise, do interpolation according to option Upscaler for img2img on decoded RGB image in pixel space
    • None or Lanczos: resize with PIL Lanczos method
    • Nearest: resize with PIL Nearest method
    • any other GAN model
      • call it only if target size is larger than firstpass either in width or height
      • check model output size, if not exactly matches target size, resize again with PIL Lanczos
• (launch img2img, sampling step count is the same as in txt2img step)

Now it becomes like this:

• (launch txt2img, generate latent images sized width/height)
• resize RGB image or latent image to according to Upscale by in both width/height
  • if hires.fix option Upscaler is in the dict latent_upscale_modes copied below, do interpolation on latent image
  • otherwise, do interpolation on decoded RGB image in pixel space
    • NOTE: hence option Upscaler for img2img in "Settings" tab no longer affects txt2img
• (launch img2img, sampling step count is the same as in txt2img step)

# modules/shared.py

latent_upscale_default_mode = "Latent"
latent_upscale_modes = {
    "Latent": {"mode": "bilinear", "antialias": False},
    "Latent (antialiased)": {"mode": "bilinear", "antialias": True},
    "Latent (bicubic)": {"mode": "bicubic", "antialias": False},
    "Latent (bicubic antialiased)": {"mode": "bicubic", "antialias": True},
    "Latent (nearest)": {"mode": "nearest", "antialias": False},
}

Hence it depends how your old pictures are upscaled, if

• your old images are center cropped in legacy version, i.e. AR of firstpass width/height and width/height mismatch
• your old resize factor cannot be computationally converted to the current Upscale by

these settings would possibly regrettably unreproducible in the latest commit...

If to try to get old results back, remember this setting mapping:

"Upscale latent space image when doing hires. fix" checked
  => set "Upscaler" to "Latent"
"Upscale latent space image when doing hires. fix" unchecked
  => set "Upscaler" to the old "Upscaler for img2img" option

[E-16]

@Kahsolt
People dont like new method and all they want is just roll everything back, but they also dont understand what the real problem is (due to too many things happening at once).

I like new method, its much more understandable, and in code way is simpler, but I also understand a problem this issue has, and I dont want to roll back anything, just want to modify what is already there, to make it done that everyone will be happy.

New method has only one serious drawback, and this is lack of the creation of a SQUARE firstpass layer image (for which all models was trained and generate best results for this specific ratio) for a NON-SQUARE upscaled image.

What we have now:

1. First pass:
   • set image canvas size
     • get Width and Height sizes
   • generate image
2. HighRes pass:
   • set image HighRes canvas size for HighRes pass:
     • get Upscale by value and multiply it by start sizes values: Width * Upscale by and Height * Upscale by
   • generate image

What we want to see (what will fix this issue):

1. First pass:
   • set image canvas size
     • get Width and Height sizes
   • generate image
2. HighRes pass:
   • set image HighRes canvas size for HighRes pass:
     • get HighRes Width and HighRes Height sizes and set HighRes canvas size by this values
   • generate image

So whats the difference? Well, people will be able to manage different ratios for different layer passes (which will return full backward compatibility)

The difference seems not that big, but on code level, there is alot, even the crop method needs to be returned.

[Kahsolt]

What I would want to see is something like this it becomes a little messy, arent it... But since these settings are inside Highres. fix, therefore they wont hardly affect the interface, I think these settings are more pleasant for people to see

PS: I just realized something, it's no longer a Highres. fix (including last update too), more like HighRes applyer or just HighRes

Certainly I love the new configuring way and you've shown the best solution that I could imagine.
I guess AUTOMATIC1111 's new ui design mainly aims to make components looking more compact.
Hours ago AUTOMATIC1111 made many subtle changes, PRs may be already on the way.
I could make a PR separating width/height back tomorrow, if AUTOMATIC1111 or anyone not yet get to ;)

I would try to allow set height/width = -1 meaning keep linking with the other one. Might this make both SQUARE party and NON-SQUARE party happy :lolipop

[E-16]

@Kahsolt
I hope you get main point that when trying to fix, we dont need anymore a Firstpass width and Firstpass height, they went into main resolution settings, instead of Upscale by we want HighRes Width and HighRes Height inside Hires. fix (or at this point more like HighRes upscaler or just HighRes)

[E-16]

Well, I guess it got fixed, in a really interesting way...
@GalaxyTimeMachine @Rogal80 @RomanescoAI try last commit 8149078 to reproduce your old pictures

[a-l-e-x-d-s-9]

Anyone who tried and survived it, please share!

[DrMeo]

I think we all got confused because we were used to the (wrong) way WebUI did things. That said, it's regrettable this change broke Sonar (for now).

WebUI used to force you to work backwards. Width and Height were previously the dimensions of the final canvas.

You mean like every other part of the UX where you put in the image size you want? txt2img, img2img, inpainting, and everywhere else you set the targets to be the final image size you want.

@EdHerdman This lunacy is like having to put in a random canvas size in txt2img and putting in a scale amount to get your final canvas size. The argument of GPU compute time being "wasted" on generating the 512x512 image to be cropped in the old way is just as ridiculous. Oh no, I lost < 0.76 seconds generating this image due to unneeded GPU compute!

The only other area that was different was working with the SD upscale script where you had to make tile sizes to generate instead of final canvas size.

[Kahsolt]

@DrMeo There are still supporters who do not think of "The argument of GPU compute time on generating the SQUARE initial image to be cropped in the old way is so ridiculous". They argue that:

• In older versions, wisely configuring your firstpass width/height and width/height all valued times of 8 will absolutely avoid center cropping and any "GPU computation waste".
• In order to get some certain really good output we use to have, cropping step is time-wasting however necessary. You could see my experimental size-travel on how image size effects the generated content. Size vary is no like seed vary, it's quite quite sensitive and non-continuous.

No bother, this functionality comes back in recent commits ;) though with more confusing wired ui options...

[Kahsolt]

Now the implementation is like:

if both "Resize width to" and "Resize height to" are 0:
    use "Upscale by"
else:
    if either "Resize width to" or "Resize height to" is 0:
        adjust the other to match aspect ratio of "width/height"
    resize while keeping aspect ratio to match the shorter side of "resize width/height"
    crop if the longer side overflow

and a more weird change in commit 99b67cf :

# special case: the user has chosen to do nothing
if self.hr_upscale_to_x == self.width and self.hr_upscale_to_y == self.height:
    self.enable_hr = False
    self.denoising_strength = None
    self.extra_generation_params.pop("Hires upscale", None)
    self.extra_generation_params.pop("Hires resize", None)
    return

hence if the actual computed upscale factor is exactly 1.0, hires.fix will refuse to do upscaling, also refuse to launch the further imgimg now 🤔…… This might be why @RomanescoAI now got mixed results?

[Oceanswave]

I think I understand why this change was made -- ostensibly to support hires fix at native 768 (and other) models. But having to maths out the correct ratio or source/target image dimensions given the model native size is a bit much.

Previously if I wanted a final resolution of 1024x768, I'd just enter that in. badda boom. That gets me the 'fix' - meaning generating a resolution closer to the trained image resolution of the model, by default of 512x512 to the entered aspect ratio- and then upscale appropriately to the input resolution.

Now I need to know that my model is 512, maths it out that the smallest variation of resolution to 512x512 for my target resolution/aspect ratio is 640x448, and set my final resolution to be 1024x768. It's a bit much.

Could not there be a way to enter the model native resolution (be it 512 or 768, etc) and have do the maths for me as it did previously?

[Edu2014]

before the update (when they took AUTOMATIC1111 from github)

now:

:(

[vsp245]

Hey guys. Hires.fix from txt2img stop wirkin for me today in Google Colab, but only for big target size res (832x1248).

1 New clean install, !git clone etc
2 Don’t change anything, I don’t touch any settings, an empty prompt.
3. Click "Hires.fix", set target size up to 1248 or "by" 2.5-3

That's all.
The new image is generated and appears in the folder, but the picture does not appear in the WebUI, moreover the WebUI no longer responds to pressing the "Generate" button - only a gradio refresh helps.

If the image is slightly smaller, around 1000px, then everything is fine. I assume that this may be due to the loading of the jpg preview to gradio.live. Anyway, changing the "JPG quality" in settings did not help.

Should I create a new bug report because of this, or is it all related to this branch?

[catboxanon]

Closing as duplicate of #6725 (probably should have been the other way around but I did not see this until now). The old functionality was restored shortly afterwards. See the wiki for more info. https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#hires-fix