Add AgX tonemapper option to Environment #87260
Conversation
Calinou
force-pushed
the
tonemap-add-agx
branch
from
8ca7b7d to
4217f1e
Compare
Calinou
force-pushed
the
tonemap-add-agx
branch
from
4217f1e to
7e101f9
Compare
Calinou
force-pushed
the
tonemap-add-agx
branch
from
7e101f9 to
37ec580
Compare
|
Maybe not related to this pr, but why does the red light for example, leak through? 
|
That's just due to how the reflection probe is set up in the scene (one probe near the red box covers the entire scene). It's not related to this PR 🙂 |
|
Hi. What version of AgX is being implemented? Troy's / or Eary's branch (the one included in blender) 
|
This relies upon the implementation found in this article, which I believe is closer to Troy's version (but not 100% identical). |
|
Thanks for the clarification. You are right, if you compare the example of the sweeps (from Troy) with the one on the website you shared, they are similar, only the red seems more saturated than in the original implementation, and since in the code the punchy version has an increase in saturation of 1.4 I think that can end up creating unwanted clipping, and it seems to be messing with the luminance values?. |
|
Not sure if it's helpful but you can also look at three.js's implementation as reference: BTW the term "white point" has a very specific meaning in RGB color, it means the chromaticity of R=G=B, in a lot of colorspaces like Rec.709, sRGB, Rec.2020 etc. it's D65, for other spaces like DCI-P3, they have their own white points like the DCI "theatre" white point. So when the PR says something about white point being 16, it looks very confusing. |
Godot doesn't have color management, so it uses an arbitrary whitepoint unit. Higher values result in less blown out highlights, but will slightly darken the whole scene. There are diminishing effects to increasing the whitepoint value, so usually, a value like 10 will look pretty close to something like 20. |
My point is, "white point" is a widely accepted language for chromaticity, I.E how warm, how cold should the white be, not brightness or intensity. A white point value is usually a CIE XYZ or CIE xyY coordinate (For example, D65's CIE XYZ is [0.95047, 1, 1.08883]), not a scalar value. If Godot uses a scalar value to refer to an intensity value and then call it "white point", the terminology is confusing. As for the white clipping point, how steep the curve is also affects the rate of change in gradients. It's best to test against EXRs like Red Xmas etc. to make sure things are still smooth. |
|
@Calinou Greetings! Is the feature gonna be in 4.3? Would be nice to have alongside with the new Global Illumination to achieve overall proper lighting and coloring in complex scenes. |
I can't give an ETA for merging this, as this PR still needs a review from other contributors before it can be merged. |
|
Google's Filament uses slightly different matrices. It says that the matrices are taken from Blender's AgX implementation. Below are the differences between the implementation in this PR and the Filament (Blender) implementation using a custom software renderer as an example. There is almost no difference, but the Filament (Blender) implementation gives a slightly darker image.
|
There was a problem hiding this comment.
I absolutely love the AgX Punchy tonemapping. I have merged this pull request in my custom build and have been using it in compatibility mode in VR with the meta quest 2.
I'm a big fan of cartoony saturated colors, and I always need to do color correction after the fact with a LUT or with the "adjustment" post processing to adjust contrast/saturation, with the AGX Punchy tonemapping I feel I don't even need further color correction, I'm getting pretty much the colors I want right away. I'll be using this in all of my projects moving forward, thank you!
All images taken with Exposure 0.7 and White 10 in Compatibility Mode with the Compatibility Mode Glow PR
| Tonemap | Far view | Close up |
|---|---|---|
| Linear |  |
 |
| Reinhard |  |
 |
| Filmic |  |
 |
| ACES |  |
 |
| AgX |  |
 |
| AgX Punchy |  |
 |
|
Thanks! I really enjoy how "natural" AgX looks, as I did not like how ACES compressed the dark colors. This is really noticeable when using global illumination. AgX seems a bit brighter than the others. AgX punchy is a bit too punchy for my use-case (I'm sure it looks better after tweaking the assets for a bit.)
(Car model CC-BY Link to Sketchfab ) |
|
It would be really nice if you guys could prioritize this for 4.3 Tonemappers affect the whole asset production pipeline, so with this we could go ham on making the assets without worrying that they'll look different later when upgrading to the release build. |
There was a problem hiding this comment.
Did a quick test, big fan of the results. AgX is doing a better job at compressing bright values compared to ACES
Exposure was adjusted to get approximately the same look (AgX needs about twice as much exposure compared to ACES)
With ACES it's always a struggle for me to not get overexposed values (notice how in AgX the pavement isn't overexposed while the shadows still retain the details):
|
Comparison on Tonemapping a HDRI:
(Used 6.0 for white in Godot)
|
This feature is also important for any global illumination upgrades since it will help a lot to achieve properly exposed areas (with more accurate colors in both dark and lit areas). Currently it's nearly impossible to achieve with existing modes. |
Want to comment on this. Originally Troy's original Punchy included a boosting of "CDL Saturation", on top of the original AgX that DID NOT HAVE OUTSET MATRIX!. And afterwards in later edition, we added the ouset matrix to the Base AgX, then in Troy's version, "Punchy" was then completely removed. (Note there we also designed a "rotation" included in the inset/outset matrix, reason will be stated later in this post.) I choose to add back the Punchy look, but since the boosting of "CDL Saturation" has already been replaced by the outset matrix, it's not needed anymore. So in the version I submitted to Blender, the Punchy look is a simple darkening without the boosting of "CDL Saturation". The original Punchy was a simple 1.35 power curve after AgX Base formation (which darkens the image), but due to OCIO's constrain, OCIO Looks are required to be pre-AgX-Base-Formation, so I had to move it to AgX Log pre formation, I tried to use some different curves to achieve in the final image the approximately the same middle grey and roughly the same "black level" as the post formation 1.35 power curve, though of course not going to be completely identical. If the doubling of outset + "CDL Saturation" is what you folks think looks better, then feel free, but I have to stress the importance of checking out those EXR files I linked up there, please make sure after your modifications that those gradients in those EXRs are still smooth as always! Too much post formation chroma boosting can lead to unsmoothness, as some boundary condition might be triggered.
I advise either go make your own rotation matrix by testing against many different challenging EXRs, or use the same one right there, it's for some compensation for Abney Effect etc. (note it's not a fix, it's currently impossible to fix Abney Effect, but the rotated matrix was tested by myself to at least somewhat compensate for it.) (Also note that the matrix was supposed to be used in BT.2020 formation space. I have already mentioned this in the three.js PR page I linked here earlier.) |
|
Using the EXR EaryCrow provided, you can see something is off with Godots AgX. |
|
Blender Cycles CPU viewport clips negatives. Better view it in compositor. Just click use nodes, drag and drop exr, and shift ctrl left click. Or at least use Cycles GPU viewport. |
There is an argument to be made that unshaded materials should match, as these do have a real function in both games and authoring in Blender for some art styles. Also, "Standard" tonemap in Blender compositor results in the exact same render as Godot's "Linear" tonemap, so I presume you must be referring to something specific of AgX when you say that Blender Compositor does a few things we just can't replicate. I agree/believe that if Godot is not matching Blender Compositor, then it should at least match some other major implementation out there, as to not introduce yet another AgX "look". Targeting an authoring tool still sounds like a good idea to me, but Blender Compositor might have been the wrong choice because it's not what is used during authoring(???) -- I'm going to ask around the dev groups I'm a part of. (Personal note: I'm baffled by how AgX is so non-standard that unshaded renders look different with every implementation, even when using the same input and output colour space?? This is bananas to me. Even trying to find the "source" for AgX seems difficult. 😕 Is there some main project page for what people mean when they say "AgX"? If so, should I make some reference renders based on this reference implementation? Or is Blender's Compositor understood to be a solid reference implementation for rec 709 input and ouptut? I have many questions, but I guess most of them are not helpful/relevant...) Edit: I should also mention that I believe latest version of this PR is the best it's ever looked, so I don't want to dismiss all the effort and iteration here. I also like the idea of removing AgX Punchy and letting users get this effect with other color correction tools. The change to perform calculations in Rec 2020 instead of Rec 709 seems like an important fix (and presumably AgX is "supposed" to be calculated in Rec 2020??). |
|
@allenwp The source of AgX is the OCIO config file found here: https://github.com/sobotka/AgX. However, the way it has been picked up is similar to how we use the term "Filmic" when describing tonemappers. Every piece of software has a "filmic" setting, but they all work slightly differently, but have the same goal "tonemap things in a way that looks good". AgX intends to be the same thing. EaryChow explains the fundamental problem quite well above. AgX is something that has to be tuned to your expected content. It isn't a "one size fits all" solution. In order to avoid certain unwanted artifacts the Blender folks added a few modifications including one they call "Guard rails" which requires doing multiple passes over the entire texture to identify the minimum luminance of the image and then use that to modify the image. This is something that is very expensive for real time game engines and would make AgX extremely costly (potentially multiple ms of frame time for average devices). That is a cost you can accept for an offline render, but not for real time. So Filament and ThreeJS opted to not use that technique. However, by not using that technique we are unable to exactly match the output of Blender exactly. Finally, with respect to color space. The original AgX uses Rec. 709, but Blender uses Rec. 2020, so they modified it to use Rec. 2020. If we want to match Blender, we need to use Rec. 2020 as well. To maybe help your confusion, colour transforms are becoming increasingly complicated in software. We used to make drastic over-simplifications for the sake of performance which had the side effect of making it very easy to ensure identical results across software. But as our knowledge of colour transformation becomes more sophisticated as an industry and as consumers of products want higher and higher quality, our colour transformation solutions have gotten more sophisticated as well. So, for the same reason you wouldn't expect a render to look the exact same between two pieces of software, you can no longer expect that a given colour transformation will look exactly the same. Each piece of software is making assumptions and taking shortcuts based on it's unique performance/quality tradeoff. |
|
Cool, thanks for clarifying and reiterating. I heard back from one artist about the workflow they'd want, and their art style requires viewing in-game-engine to see what things look like. They don't preview any materials in Blender. My only concern with Filament/threejs's approach (this PR's current version) is the following issues with shifting towards R, G, and B values prematurely when approaching black. This doesn't seem to happen with either Blender's approach or Godot's approach when calculating in rec 709 space:
Image file links: (On some monitors it's possible to see a pure red curve that eats into the dark oranges, instead of a straight line. This is because G and B values hit zero much earlier when approaching black, relative to the R value. This is clearly evident when dragging an eyedropper tool left and right in an image editing app.) |
I think I need to look at these on a different monitor as I'm not able to see that. I'm currently looking on a MBP screen and I don't see any notable hue shifting differences between Blender and Godot with Rec. 2020. Godot with Rec. 709 has that noticeable hue shift towards yellow. Tried on a different monitor and still can't notice any hue shifting |
Here's what I was referring to. It seems that it hits pure red at a red value of around 90, while Blender is around 20 and rec 709 is around 40 or 50. On my TV it's visible, so maybe this is sort of issue is a problem with consumer displays rather than professional monitors. This is why I'm going more by what the eyedropper tool gives me, but the way the image looked on my TV is really what prompted me to mention this here. Edit: I looked on my iPhone (13 Pro) and I cannot see the "hard line" that I see on my TV, but to me the dark oranges turn more red as they approach black with rec 2020 calculations than with rec 709 calculations or with Blender's approach. So this probably comes down to the rec 2020 calculations giving a different "look" that is arguably, for this case of dark oranges, less neutral than the rec 709 calculations. I'm fine with either of rec 709 calculations or rec 2020 calculations. They're different looks. And since there isn't a true reference for how AgX is supposed to behave, it's probably better to side towards filament/three.js. The earlier shifts to pure red, green, and blue when approaching black and overall colour curves that result from rec 2020 calculations is arguably going to be preferred by most Godot users(???) because it makes the image look less washed out. |
|
@allenwp take a look at the comparison between Threejs clipped in 709 vs clipped in 2020 above from PavielKraskouski above #87260 (comment) The current draft of this PR is clipping in 2020 because it retains a level of saturation much closer to Blender then when doing clipping in 709. However, looking at the dark sections in those images, I do see quite a bit more red in the Rec. 2020 version. In either case though, I think clipping in Rec. 2020 looks much closer to blender than clipping in Rec. 709. So that might just be a tradeoff we have to accept |
|
Let me know if some three.js or filament comparisons would be beneficial before the merge. Based on the one three.js render from PavielKraskouski, this rec 2020-based PR appears virtually identical to three.js with rec 2020 clipping, as expected. |
There was a problem hiding this comment.
As discussed here and on Rocketchat, we've reached a place where our AgX implementation looks quite similar to Blender and to other open source engines (Threejs, Filament), but doesn't quite match them.
It is unlikely that we will exactly match Blender since they do some full-image processing that is not feasible in realtime.
We could exactly match Threejs and Filament by clamping color in Rec. 709 space before converting to Rec. 2020. However, clamping in Rec. 2020 looks much more similar to blender, and avoids the strong desaturation that we had previously, and that Threejs and Filament have.
At this point we know that we can't match Blender, so our discussions are starting to circle around what an ideal implementation would look like, but the nature of AgX is that it will always be a stylistic choice. There is no 100% "correct" way of implementing it. So what we need at this point is to get this into the hands of more users and get their feedback. We can easily make changes before beta if artists are unhappy with the look of the current implementation.
|
I discovered that this PR was converting to linear space using pow(2.2), like the original IOLITE post... But EaryChow's uses a 2.4 exponent. Here's a comparison of this PR (with 2.2) and a modified version with |
|
Should I switch this PR to use |
2.4 looks significantly closer to Blender (even it's not 100% there) |
|
@PavielKraskouski Produced the following comparison between his tonemapping program and Blender using 2.4 and 2.2 gamma https://imgsli.com/MzI2ODg5/0/1 Then he shared his source on RC so we can get ours looking the same. Here is a patch. Using this, our results match the imgsli above set to "2.4". It brings us extremely close to Blender edit: Turns out I had made a mistake in some earlier tests. The change to 2.4 was the only needed change. The patch also modifies the outset matrix, but all it does is bake the IMO with this patch this should be ready to merge now!! |
Co-authored-by: Clay John <claynjohn@gmail.com>
Calinou
force-pushed
the
tonemap-add-agx
branch
from
3f2b943 to
87e3d1e
Compare
|
I've pushed an update with the above patch incorporated. Here's a comparison: Saturation 1.0
Saturation 1.3
|
|
Quick question: Do the adjustments, like saturation, apply to a high bit depth buffer, or does it apply to a tonemapped 8bit per channel image? (And thus, information is lost when applying adjustments.) |
|
@WrobotGames If I understand the code correctly, adjustments are applied after tone mapping and gamma correction to normalized floating point colors. Here I have a question. Maybe it is better to do this with linear color values? Also, I noticed that when adjusting saturation, color luminance is calculated as the average of RGB. It is better to use luminance coefficients (0.2126, 0.7152, 0.0722), in my opinion. These coefficients work in linear sRGB. |
They are done in the same shader, so they are still using 32bpc channel colours. However, adjustments happen in sRGB gamma space. So they are not linear. You won't have 8-bit artifacts though |
|
Okay good, I was afraid of 8bit artefacts. |
|
There is room for a lot of valuable comments in the AgX shader code. Both Filament and EaryChow's code has examples of these sorts of comments, like how matrices were generated, how constants were calculated, etc. It might be fine to just point to them, but it takes a bit of digging to get an understanding as-is. Now that the matrix inversion has been baked in, it makes it a little harder to follow the history. All that to say, I might make some comment suggestions tomorrow as a github review, but don't have time to do so tonight. |
For context, the matrix comes from the Threejs PR mrdoob/three.js#27366 and it is just a pre-calculated version of the one used by Eary. I thought I had copied Threejs, but I actually copied bits from Threejs and bits from Filament. If you add a comment, you can just point to Eary's matrix and say that this has the inverse baked in |
|
I noticed that EaryChow's implemenation that Blender uses applies the Adjusting this PR to do the same as Blender, unsurprisingly, makes it look more like Blender's render: I'm sorry I didn't have time to dig deeper into this code before now. Would you mind if I took a few days next week to continue to dig into this? |
|
@allenwp Go ahead! The other two major differences from blender are:
|
|
Thanks, @clayjohn. Those are the only two intentional differences I’ve found, as well. Next step is to verify the sigmoid LUT that the polynomial approximation was based off of is actually the same sigmoid LUT that Blender uses and that the constants that have been copy-pasted around from IOLITE have been calculated in a way that matches EaryChow’s… I’ll post back here soon enough :) |
|
@allenwp If you want to try out the chroma rotation that Blender does, I made a PoC yesterday. I wasn't happy with the results. But this is that second major difference between ours and theirs |
Tried it in my branch and it seems to prevent shifts to the notorious six, make images look closer to Blender, and subjectively make the test images look better. Doesn't address the other brightness curve issues and loss of bright saturated greens I'm still looking into, so maybe those are related to the constants/sigmoid approximation or maybe they're related to the guard rail. Obviously, the chroma rotation is expensive, so not sure if users would want this tradeoff to get less hue shift and get closer to Blender. Be better to decide that when I've finished looking into the rest. |
|
Summary of our progress on implementing a Blender AgX approximation into Godot so far: There are two root AgX implementations that are most relevant to this discussion: the original AgX, created by Troy Sobotka (referred to here as Troy’s AgX) and a fork of Troy’s AgX created for Blender by Eary Chow et al. (referred to here as Blender’s AgX). Neither of these were implemented for realtime/game rendering; both were implemented as OpenColorIO formats. Blender’s AgX uses a cube LUT that is generated by a python script that is found in EaryChow’s repository. IOLITE published an implementation of Troy’s AgX that is suitable for realtime rendering by making use of a polynomial approximation of the sigmoid contrast curve. Since we’re interested in matching Blender’s AgX rather than Troy’s, I haven’t bothered to review it in depth and can’t speak to the accuracy of its implementation. Google Filament and three.js both appear to have been implemented using IOLITE’s AgX as a starting point with modifications to take some, but not all, elements of Blender’s AgX: calculations in the Rec. 2020 color space and Blender’s inset and outset matrices. This results in an implementation that is neither Troy’s AgX nor Blender’s AgX. The renders appear washed out and brighter when compared to Blender’s AgX. This Godot PR followed the approach used by Google Filament and three.js and had the same differences compared to Blender’s AgX and Troy's AgX. PavielKraskouski, clayjohn, and I have been talking in RocketChat and are effectively in the process of re-implementing the shader in Godot to use Blender’s AgX as a starting point, rather than basing it off of IOLITE, Google Filament, three.js, or Troy’s AgX. Progress has been very good so far and it looks like we will be able to match Blender’s AgX more closely than previous efforts. |
|
I've completed rewriting the AgX shader based on the original Blender/EaryChow implementation rather than based on the Filament/three.js versions that incorrectly combined elements of Troy's and Blender's configurations. Through the process of rewriting the shader, I made the following corrections, compared with the current state of this PR:
Additionally, I improved performance relating to the matrices by combining the AgX outset matrix with rec 2020 matrix. Comments have been added to better describe the choices made in this approach. Thanks to clayjohn, PavielKraskouski, Calinou, and others for helping me out and talking me through this. I have created two branches that are ready to be merged into this branch and duplicated into the compatibility renderer:
Blender AgX (no hue rotation) in Godot
Here is my standard table that compares these Godot implementations with "ground truth" Blender. Differences to look out for is reds shifting to yellow (or not shifting to yellow). Also, note that I was not able to perfectly match the contrast curve that Blender uses. I tried my best... Given more time I might be able to get closer, but polynomial regressions are a finicky thing! I also made a branch with a number of AgX implementations that you can easily switch between to preview the different versions, including a low performance Blender reference implementation (that includes Blender's "lower guardrail" and a high-order polynomial approximation of the "sigmoid" contrast curve) and Tory’s AgX. I'd love to spend more time discussing this and choosing which of these is implemented into Godot and how hue shifts to the notorious six (specifically yellow) are addressed in Godot, but I'm off on holidays starting tomorrow and will be back on Jan 13. |
Thanks to @begla for providing this code under MIT license and helping with the whitepoint configuration 🙂
Testing project: godotengine/godot-demo-projects#857
TODO
Preview
All images use whitepoint 6.0 unless otherwise mentioned.
Footnotes
This demo is currently not designed with Compatibility in mind, so the surface colors are incorrect. Nonetheless, the tonemappers work. ↩