Implement LightmapGI shadowmasks

[BlueCube3310]

Salvages #77284
Closes: godotengine/godot-proposals#2354

Adds support for baked shadowmasks to LightmapGI. The shadowmasks are saved as compressed images and applied on top of the dynamic shadowmaps.

Progress:

• Support baking shadowmasks,
• Support Forward+ renderer,
• Support Mobile renderer,
• Support Compatibility renderer,
• (Bugfix) Allow lightmaps without shadowmasks,
• Dilate,
• Fix blending artifacts, Edit: Caused by DXT1 Compression,
• Document everything properly,
• Clean up and optimize the code,

To consider:

• Consider using specialization constants,
• Consider blending seams.

To do in the future (important):

• Compress grayscale textures as RGTC_R, Edit: done
• Implement and use bicubic sampling (See Implement bicubic sampling for lightmaps #89919) Edit: done,
• Further optimizations and cleanup,
• Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.
• Remind the user when lighting needs to be rebuilt,

[AThousandShips]

Since this is salvaged from a previous PR, unless it doesn't contain any of their code, please add them as co-author

[BlueCube3310]

Did I do it correctly? Sorry, I'm still new to github.

[BlueCube3310]

I think it's ready for review now. I'm still not certain whether the documentation is clear enough, or if the seams are noticeable enough to require blending them, though.

[Calinou]

Filter the shadowmasks in the Lightmapper,

Reimplementing support for bicubic sampling could resolve this. If this doesn't suffice, we could add an option that uses Image::resize() to halve the shadowmask resolution, then use Image::resize() on the small image to bring it back to the original size with cubic resampling. This will effectively blur the shadowmask without requiring an expensive filter.

Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.

This is a clever idea I hadn't thought of before. It makes a lot of sense if you need sharp distant shadows but don't need indirect lighting data to be very precise 🙂

That said, in practice, you don't want distant shadows to be too precise as you run the risk of having aliasing visible in the distance (lightmaps and shadowmasks aren't mipmapped).

[Calinou]

Tested locally, it mostly works as expected (I tested Forward+ and Mobile).

Testing project: test_lightmapgi_mobile.zip

The way shadowmask interacts with real-time shadows looks strange. I don't recall this happening with previous PRs, but maybe I started noticing this issue just now. This is most noticeable with Directional Soft Shadow Filter Quality set to Ultra, but it occurs with any setting:

I'd expect the final shadow color to be min(shadowmask, real_time_shadowmap). Right now, it seems to be real_time_shadowmap > 0.0 ? real_time_shadowmap : shadowmask. This issue affects both Forward+ and Mobile.

If you set Directional Soft Shadow Filter Quality to Hard, notice how shadowed area transitions look much sharper than intended:

Regarding fixing seams, I'd say it's worth the effort as right now, seams can be quite noticeable in the distance:

Real-time shadow	Shadowmask

[BlueCube3310]

Done, this should now work as suggested

[BlueCube3310]

It looks like now a line appears where the shadow splits should end (when using blended shadows), but that also happens on the master branch and is unrelated to this PR

[SpockBauru]

Tested my MRP on Forward+, Mobile and Compatibility renderers. Shadowmasks is working as intended without any visual issues. Baked the LightmapGI on each renderer and everything is working.

Also made a quick test on my (halted) personal project. Out of the box, it works as expected without any issues. Far shadows are now present:

Shadowmaps None	Shadowmaps Replace

With this developers can make an option to reduce shadowmaps resolution and distance, improving the performance on weak systems.

I didn't find any issue related to this PR and believe that is ready to merge.

[clayjohn]

Let's go ahead and merge this!

It looks great and while I think there is room for iteration, I think it's time to get this in the hands of users so we can get some feedback

[akien-mga]

Thanks!

[Calinou]

@BlueCube3310 I noticed shadowmasks are saved as PNGs, but saving them as lossless WebPs can make them much smaller on disk. Example with a 4096×4096 shadowmask texture:

321k test_shadow.png
151k test_shadow.webp

This does not affect memory usage, but it'll reduce file sizes in version control, which is helpful as lightmap data is often iterated upon. In the codebase, it should be as simple as replacing save_png() to save_webp() (which is lossless by default).

The downside is that we can only do that for images that are 16383×16383 or smaller, so shadowmasks that are larger would have to be resized down. In practice, only 16384×16384 can be an issue here¹, so we could downsize it to 16383×16383 upon saving (Godot's VRAM compression will pad the last pixel automatically).

Footnotes

1. The maximum lightmap image size that can be saved by Godot is 16384×16384. ↩

[michaelharmonart]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

[clayjohn]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

That's an existing issue in LightmapGI. #98684

[michaelharmonart]

ahh ok. l never noticed since there was never a way to blend between real-time and baked lighting like this before now

I noticed now only since there was a clear transition line between realtime and baked shadows only since there was a mismatch in which objects had shadows.

[IsaacMarovitz]

In 4.4.dev7 despite having shadowmask mode set to Replace, baking does not seem to build the shadowmasks.

I can't tell if I'm using it wrong, or if this is a bug.

[SpockBauru]

baking does not seem to build the shadowmasks.

@IsaacMarovitz It's working here, did you remembered to set shadowmasks to replace before baking?

[IsaacMarovitz]

Nvm, I was using it wrong, my directional light was set to static and I missed the warn in the console. The DirectionalLight Bake Mode documentation needs to be updated to reflect this change.

[unfa]

Hey, where can I find a good explanation of this feature? I was looking around and couldn't really find anything.

[SpockBauru]

Officially there's an explanation on the "latest" version of the docs, not the "stable" one since 4.4 stable was not released yet. Be aware that it's an experimental feature so there's no much explanation yet:

• https://docs.godotengine.org/en/latest/classes/class_lightmapgi.html#class-lightmapgi-property-shadowmask-mode
• https://docs.godotengine.org/en/latest/classes/class_lightmapgidata.html#enum-lightmapgidata-shadowmaskmode

I'm also answering people on this Reddit thread: https://www.reddit.com/r/godot/comments/1hcr73g

[Calinou]

Hey, where can I find a good explanation of this feature? I was looking around and couldn't really find anything.

I've opened a PR to document shadowmasks: godotengine/godot-docs#10482

[GustJc]

Not sure if something changed.
But using the latest master (v4.4.dev.gh [24d7451]) and trying to bake the lightmapGI using the testproject from (#85653 (comment)) crashes Godot.
With a lot of "Couldn't map PC to fn name" errors.

But testing it using a simple scene on a new projected or using the test_lightmapgi_mobile project (#85653 (review)) worked fine.

shadowmap_bug.mp4

Here is the start of the crash:
(On Windows 10, Foward+, Nvidea 1070)

[SpockBauru]

Compiled the master branch locally and can confirm that the demo is not working anymore. First time I baked, it got my PC unresponsive for a while and I had to make a hard reset. Second time it crashed Godot.

My system: Godot v4.4.dev (24d7451) - Windows 11 (build 22631) - Multi-window, 1 monitor - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3080 (NVIDIA; 32.0.15.6614) - 12th Gen Intel(R) Core(TM) i7-12700KF (20 threads)

@GustJc Would you kindly open a new issue?

@Geometror
EDIT: Tested some some PR's and is crashing since #99538 (Transparency PR). Something broke between Dec 12 and Dec 19.

EDIT2: It seems like if I insist and try again it manage to bake, but taking double the time and the computer becomes unresponsive in the process.