Fixed StyleBoxFlat antialiasing to be applied only on corners

[pouleyKetchoupp]

After multiple changes to the transparent drawing used for AA, it ended up being consistently applied all around the borders inside and outside, which is not the intended behavior. This change makes sure border flat sides are solid instead of blurry.

Fixes #35279

[akien-mga]

CC @Zylann

[pouleyKetchoupp]

@Zylann Can you check if this PR addresses #35279 for you? It should finally do the intended behavior without breaking previous fixes and without the need for decimal numbers, but since we don't get exactly the same results I'd like to make sure I'm not missing something.

[Zylann]

In Godot 3.2 beta5:

With this PR:

The outlined borders look good, but the non-outlined ones still look blurry and cause an overlap.

In beta5, everything seems to look right, but the only thing that actually looked wrong were corners of the non-outlined boxes, as they seemed to miss AA completely:

which is also what was reported originally in #34830
So I believe this is the only thing that needed fixing, however I don't have much knowledge of the code drawing this (other than it approximates AA with extra geometry).

[groud]

Does it work with rotated controls?

[Zylann]

Beta5:

RC2:

PR:

Between Beta5 and PR, same results as I described before, although you'll need to squint to see all the details on these screenshots.

While RC2 does have better results on straight rotated lines, it's quite a matter of "luck" for the rotated control (since between "blurry" line and an AAed line you wouldn't tell much difference), while real AA would have looked correct in all screenshots... (note it still has the artifact below the tab, and spacings look slightly off) at this point I'm starting to believe the way AA is currently implemented will always have drawbacks like this.

[pouleyKetchoupp]

Based on these screenshots, it looks like this PR is actually a regression for some cases.

In the RC2 screenshot:
It's not lucky it looks better when rotated. The blurry line is the way AA is implemented, and it's applied all along all borders to make sure it works in all possible cases.
The extra line under highlighted tabs and the space under non-highlighted tabs are due to the fact we're at the same time making the border slightly thinner and we're adding a blurry line to fake the AA.

I'm going to check again if I can adjust the position and thickness of the border in the case of AA to fix the artifacts visible in the screenshot from RC2 without a need for adjusting positions with decimal values.

[Zylann]

It's not lucky it looks better when rotated

Maybe not the right word, I understand what you describe^^ what I was saying is that it looks right, because it would be hard to distinguish between a blurry line and a properly anti-aliased line when it's rotated anyways, because they happen to produce similar results ("similar", because it's still blurrier than AA). The problem becomes more obvious when it's horizontal or near-horizontal, and way less visible when diagonal.
Basically, I'm pretty sure now that proper AA can't be achieved the way it's implemented, there will always be required half-pixel tweaks depending on the orientation of the shapes (horizontal/vertical vs round corners and diagonals).

I recall AA was never really supported with 2D polygons in Godot... any chance for that to happen one day? Hopefully if #12840 is fixed then all these workarounds would no longer be needed.
However I see #10241 was similar and closed as fixed, so I'm a bit confused.

[Calinou]

@Zylann There are plans to support MSAA in 2D in 4.0. That said, I think fake antialiasing still has the potential to be faster, especially on low-end hardware.

[Zylann]

At this point I was starting to investigate texture styleboxes generated by GDScript (since my app has customizable theme). It's just significantly more work to achieve almost the same thing^^

[pouleyKetchoupp]

@Zylann I agree, there will be always problems in some edge cases with the current fake AA implementation. I'll spend a bit more time this week to see what kind of compromise I can find in order to avoid obvious glitches like the extra thin line you have in #35279, but at this point I think having blurry lines all around the borders are needed to handle more cases, including rotations.

About AA in 2D:
I've made a fix in PR #33857 that enables some AA for Polygon2D & Line2D, but it has the constraint that you need to disable hdr in your project for that implementation to work.
For the long term, AA will be generally reworked for 4.0 in order to support MSAA, as @Calinou mentioned.

[pouleyKetchoupp]

PR #35579 is a better take on this issue, it has no regression on rotations. Closing this one.