Implement shadowmask for DirectionalLight in BakedLightmap

[Calinou]

This can be used to fade distant real-time DirectionalLight shadows with baked shadows (stored in the lightmap's alpha channel).

See godotengine/godot-proposals#2354 (comment) for an explanation of why I went with shadowmasking instead of subtractive shadowmapping. This implementation allows changing the DirectionalLight's color and shadow color at run-time, and does not require manual work from the user to define a hand-picked shadow color that looks good.

This closes godotengine/godot-proposals#2354.

Testing project: test_shadowmask_3.x.zip

August 2021 patch

From 6e3233483a5c564efb0ad5b4d7da7d3549b24508 Mon Sep 17 00:00:00 2001
From: Hugo Locurcio <hugo.locurcio@hugo.pro>
Date: Tue, 3 Aug 2021 02:30:48 +0200
Subject: [PATCH] Implement shadowmask for DirectionalLight in BakedLightmap

This can be used to fade distant real-time DirectionalLight shadows
with baked shadows (stored in the lightmap's alpha channel).

TODO:

- Restore support for baking lightmaps with HDR.
- Fix the resulting lightmap when denoising is enabled.
---
 doc/classes/BakedLightmap.xml               |   4 +
 doc/classes/Light.xml                       |   1 +
 drivers/gles2/shaders/scene.glsl            |  30 +++--
 drivers/gles3/shaders/scene.glsl            |  22 +++-
 modules/lightmapper_cpu/lightmapper_cpu.cpp | 115 ++++++++++++++------
 modules/lightmapper_cpu/lightmapper_cpu.h   |   9 +-
 scene/3d/baked_lightmap.cpp                 |  59 +++++++++-
 scene/3d/baked_lightmap.h                   |   4 +
 scene/3d/light.cpp                          |   5 +-
 9 files changed, 195 insertions(+), 54 deletions(-)

diff --git a/doc/classes/BakedLightmap.xml b/doc/classes/BakedLightmap.xml
index ddb59da4aba1..9114c3355d23 100644
--- a/doc/classes/BakedLightmap.xml
+++ b/doc/classes/BakedLightmap.xml
@@ -92,6 +92,10 @@
 			If [code]true[/code], stores the lightmap textures in a high dynamic range format (EXR). If [code]false[/code], stores the lightmap texture in a low dynamic range PNG image. This can be set to [code]false[/code] to reduce disk usage, but light values over 1.0 will be clamped and you may see banding caused by the reduced precision.
 			[b]Note:[/b] Setting [member use_hdr] to [code]true[/code] will decrease lightmap banding even when using the GLES2 backend or if [member ProjectSettings.rendering/quality/depth/hdr] is [code]false[/code].
 		</member>
+		<member name="use_shadowmask" type="bool" setter="set_use_shadowmask" getter="is_using_shadowmask" default="true">
+			If [code]true[/code], bakes the [DirectionalLight]s' direct light shadows into a [i]shadowmask[/i] which is stored in the lightmap textures' alpha channel. This shadowmask is used to keep static shadows visible past the DirectionalLights' [member DirectionalLight.directional_shadow_max_distance] by blending the last shadow split with the shadowmask. This in turn allows you to use a lower [member DirectionalLight.directional_shadow_max_distance] for dynamic objects. Lower real-time shadow distances improve shadow detail and performance while making shadow acne less visible.
+			[b]Note:[/b] Shadowmasking only supports one baked [DirectionalLight] at a time. If you have more than one [DirectionalLight] whose [member Light.light_bake_mode] isn't set to [constant Light.BAKE_DISABLED], the editor will print a warning when baking lightmaps.
+		</member>
 	</members>
 	<constants>
 		<constant name="BAKE_QUALITY_LOW" value="0" enum="BakeQuality">
diff --git a/doc/classes/Light.xml b/doc/classes/Light.xml
index fc3183670ae7..a0adf7a40751 100644
--- a/doc/classes/Light.xml
+++ b/doc/classes/Light.xml
@@ -51,6 +51,7 @@
 		</member>
 		<member name="light_size" type="float" setter="set_param" getter="get_param" default="0.0">
 			The size of the light in Godot units. Only considered in baked lightmaps and only if [member light_bake_mode] is set to [constant BAKE_ALL]. Increasing this value will make the shadows appear blurrier. This can be used to simulate area lights to an extent.
+			[b]Note:[/b] [DirectionalLight]s with their bake mode set to [constant BAKE_INDIRECT] still have an adjustable [member light_size] property, but it will only be used for the baked shadowmask (see [member BakedLightmap.use_shadowmask]). This can be used to better integrate the shadowmask with real-time shadows by making it softer and less pixelated. Values between [code]0.01[/code] and [code]0.1[/code] work well for the purposes of shadowmasking.
 		</member>
 		<member name="light_specular" type="float" setter="set_param" getter="get_param" default="0.5">
 			The intensity of the specular blob in objects affected by the light. At [code]0[/code], the light becomes a pure diffuse light. When not baking emission, this can be used to avoid unrealistic reflections when placing lights above an emissive surface.
diff --git a/drivers/gles2/shaders/scene.glsl b/drivers/gles2/shaders/scene.glsl
index 12a6bbb55ace..5394bf071d11 100644
--- a/drivers/gles2/shaders/scene.glsl
+++ b/drivers/gles2/shaders/scene.glsl
@@ -1779,12 +1779,13 @@ FRAGMENT_SHADER_CODE
 	}
 
 #ifdef USE_LIGHTMAP
-//ambient light will come entirely from lightmap is lightmap is used
 #if defined(USE_LIGHTMAP_FILTER_BICUBIC)
-	ambient_light = texture2D_bicubic(lightmap, uv2_interp).rgb * lightmap_energy;
+	vec4 lightmap_sample = texture2D_bicubic(lightmap, uv2_interp);
 #else
-	ambient_light = texture2D(lightmap, uv2_interp).rgb * lightmap_energy;
+	vec4 lightmap_sample = texture2D(lightmap, uv2_interp);
 #endif
+	//ambient light will come entirely from lightmap is lightmap is used
+	ambient_light = lightmap_sample.rgb * lightmap_energy;
 #endif
 
 #ifdef USE_LIGHTMAP_CAPTURE
@@ -1897,6 +1898,16 @@ FRAGMENT_SHADER_CODE
 #endif
 	float depth_z = -vertex.z;
 
+#ifdef USE_LIGHTMAP
+	// Take the DirectionalLight's shadow color into account for the shadowmask.
+	// This applies even if the DirectionalLight shadow is disabled, which can be
+	// done to improve performance by disabling shadows for dynamic objects only.
+	vec3 shadowmask = mix(shadow_color.rgb, vec3(1.0), lightmap_sample.a);
+#else
+	// No lightmaps, fallback to no shadows in the distance.
+	vec3 shadowmask = vec3(1.0);
+#endif
+
 #if !defined(SHADOWS_DISABLED)
 
 #ifdef USE_SHADOW
@@ -1960,7 +1971,7 @@ FRAGMENT_SHADER_CODE
 			shadow_att = mix(shadow_att, shadow_att2, pssm_blend);
 		}
 #endif
-		light_att *= mix(shadow_color.rgb, vec3(1.0), shadow_att);
+		light_att *= mix(shadow_color.rgb, shadowmask, shadow_att);
 	}
 
 #endif //LIGHT_USE_PSSM4
@@ -2000,14 +2011,14 @@ FRAGMENT_SHADER_CODE
 			shadow_att = mix(shadow_att, shadow_att2, pssm_blend);
 		}
 #endif
-		light_att *= mix(shadow_color.rgb, vec3(1.0), shadow_att);
+		light_att *= mix(shadow_color.rgb, shadowmask, shadow_att);
 	}
 
 #endif //LIGHT_USE_PSSM2
 
 #if !defined(LIGHT_USE_PSSM4) && !defined(LIGHT_USE_PSSM2)
 
-	light_att *= mix(shadow_color.rgb, vec3(1.0), sample_shadow(light_directional_shadow, shadow_coord));
+	light_att *= mix(shadow_color.rgb, shadowmask, sample_shadow(light_directional_shadow, shadow_coord));
 #endif //orthogonal
 
 #else //fragment version of pssm
@@ -2103,11 +2114,16 @@ FRAGMENT_SHADER_CODE
 			}
 #endif
 
-			light_att *= mix(shadow_color.rgb, vec3(1.0), shadow);
+			light_att *= mix(shadow_color.rgb, shadowmask, shadow);
 		}
 	}
 #endif //use vertex lighting
 
+#else
+	// Use the shadowmask only when past the DirectionalLight's maximum shadow distance.
+	// FIXME: This doesn't work when the DirectionalLight's shadow is enabled.
+	// It only works when the shadow is disabled.
+	light_att *= shadowmask;
 #endif //use shadow
 
 #endif // SHADOWS_DISABLED
diff --git a/drivers/gles3/shaders/scene.glsl b/drivers/gles3/shaders/scene.glsl
index 8de8b478b7da..0e6f88d691b3 100644
--- a/drivers/gles3/shaders/scene.glsl
+++ b/drivers/gles3/shaders/scene.glsl
@@ -1927,12 +1927,15 @@ FRAGMENT_SHADER_CODE
 #endif
 
 #ifdef USE_LIGHTMAP
+	// Also store the alpha channel as it's used for shadowmasking further below.
 #ifdef USE_LIGHTMAP_LAYERED
-	ambient_light = LIGHTMAP_TEXTURE_LAYERED_SAMPLE(lightmap, vec3(uv2, float(lightmap_layer))).rgb * lightmap_energy;
+	vec4 lightmap_sample = LIGHTMAP_TEXTURE_LAYERED_SAMPLE(lightmap, vec3(uv2, float(lightmap_layer)));
 #else
-	ambient_light = LIGHTMAP_TEXTURE_SAMPLE(lightmap, uv2).rgb * lightmap_energy;
-#endif
-#endif
+	vec4 lightmap_sample = LIGHTMAP_TEXTURE_SAMPLE(lightmap, uv2);
+#endif //USE_LIGHTMAP_LAYERED
+
+	ambient_light = lightmap_sample.rgb * lightmap_energy;
+#endif //USE_LIGHTMAP
 
 #ifdef USE_LIGHTMAP_CAPTURE
 	{
@@ -2011,7 +2014,15 @@ FRAGMENT_SHADER_CODE
 
 #if defined(USE_LIGHT_DIRECTIONAL)
 
+#ifdef USE_LIGHTMAP
+	// Take the DirectionalLight's shadow color into account for the shadowmask.
+	// This applies even if the DirectionalLight shadow is disabled, which can be
+	// done to improve performance by disabling shadows for dynamic objects only.
+	vec3 light_attenuation = mix(shadow_color_contact.rgb, vec3(1.0), lightmap_sample.a);
+#else
+	// No lightmaps, fallback to no shadows in the distance.
 	vec3 light_attenuation = vec3(1.0);
+#endif
 
 	float depth_z = -vertex.z;
 #ifdef LIGHT_DIRECTIONAL_SHADOW
@@ -2132,10 +2143,11 @@ FRAGMENT_SHADER_CODE
 			shadow = min(shadow, contact_shadow);
 		}
 #endif
-		light_attenuation = mix(mix(shadow_color_contact.rgb, vec3(1.0), shadow), vec3(1.0), pssm_fade);
+		light_attenuation = mix(mix(shadow_color_contact.rgb, light_attenuation, shadow), light_attenuation, pssm_fade);
 	}
 
 #endif // !defined(SHADOWS_DISABLED)
+
 #endif //LIGHT_DIRECTIONAL_SHADOW
 
 #ifdef USE_VERTEX_LIGHTING
diff --git a/modules/lightmapper_cpu/lightmapper_cpu.cpp b/modules/lightmapper_cpu/lightmapper_cpu.cpp
index e19c9096ee33..4c077c3d0514 100644
--- a/modules/lightmapper_cpu/lightmapper_cpu.cpp
+++ b/modules/lightmapper_cpu/lightmapper_cpu.cpp
@@ -804,11 +804,16 @@ void LightmapperCPU::_compute_direct_light(uint32_t p_idx, void *r_lightmap) {
 			}
 		}
 
-		Vector3 final_energy = attenuation * penumbra * light_energy * MAX(0, normal.dot(-light_to_point));
+		const Vector3 final_energy = attenuation * penumbra * light_energy * MAX(0, normal.dot(-light_to_point));
 		lightmap[p_idx].direct_light += final_energy * light.indirect_multiplier;
 		if (light.bake_direct) {
 			lightmap[p_idx].output_light += final_energy;
 		}
+
+		if (light.type == LIGHT_TYPE_DIRECTIONAL) {
+			// Store the directional shadowmask, which does not depend on the light color, energy or angle.
+			lightmap[p_idx].shadowmask = CLAMP(lightmap[p_idx].shadowmask - attenuation * penumbra, 0.0, 1.0);
+		}
 	}
 }
 
@@ -925,7 +930,7 @@ void LightmapperCPU::_post_process(uint32_t p_idx, void *r_output) {
 
 	LocalVector<int> &indices = scene_lightmap_indices[p_idx];
 	LocalVector<LightmapTexel> &lightmap = scene_lightmaps[p_idx];
-	Vector3 *output = ((LocalVector<Vector3> *)r_output)[p_idx].ptr();
+	Color *output = ((LocalVector<Color> *)r_output)[p_idx].ptr();
 	Vector2i size = mesh.size;
 
 	// Blit texels to buffer
@@ -934,7 +939,11 @@ void LightmapperCPU::_post_process(uint32_t p_idx, void *r_output) {
 		for (int j = 0; j < size.x; j++) {
 			int idx = indices[i * size.x + j];
 			if (idx >= 0) {
-				output[i * size.x + j] = lightmap[idx].output_light;
+				output[i * size.x + j] = Color(
+						lightmap[idx].output_light.x,
+						lightmap[idx].output_light.y,
+						lightmap[idx].output_light.z,
+						1.0 - lightmap[idx].shadowmask);
 				continue; // filled, skip
 			}
 
@@ -966,7 +975,11 @@ void LightmapperCPU::_post_process(uint32_t p_idx, void *r_output) {
 			}
 
 			if (closest_idx != -1) {
-				output[i * size.x + j] = lightmap[closest_idx].output_light;
+				output[i * size.x + j] = Color(
+						lightmap[closest_idx].output_light.x,
+						lightmap[closest_idx].output_light.y,
+						lightmap[closest_idx].output_light.z,
+						1.0 - lightmap[closest_idx].shadowmask);
 			}
 		}
 	}
@@ -983,23 +996,57 @@ void LightmapperCPU::_post_process(uint32_t p_idx, void *r_output) {
 		Ref<LightmapDenoiser> denoiser = LightmapDenoiser::create();
 
 		if (denoiser.is_valid()) {
-			int data_size = size.x * size.y * sizeof(Vector3);
-			Ref<Image> current_image;
-			current_image.instance();
+			// FIXME: This denoiser handling doesn't work as expected.
+
+			const int data_size_rgb = size.x * size.y * sizeof(Vector3);
+			Ref<Image> current_image_rgb;
+			current_image_rgb.instance();
 			{
 				PoolByteArray data;
-				data.resize(data_size);
+				data.resize(data_size_rgb);
 				PoolByteArray::Write w = data.write();
-				memcpy(w.ptr(), output, data_size);
-				current_image->create(size.x, size.y, false, Image::FORMAT_RGBF, data);
+				print_line("Creating RGB image");
+				memcpy(w.ptr(), output, data_size_rgb);
+				current_image_rgb->create(size.x, size.y, false, Image::FORMAT_RGBF, data);
 			}
 
-			Ref<Image> denoised_image = denoiser->denoise_image(current_image);
+			Ref<Image> denoised_image = denoiser->denoise_image(current_image_rgb);
+
+			// Store the alpha channel (shadowmask) separately as OIDN does not
+			// support denoising the alpha channel. Moreover, the shadowmask
+			// does not require any form of denoising.
+			const int data_size_a = size.x * size.y;
+			Ref<Image> current_image_a;
+			current_image_a.instance();
+			{
+				PoolByteArray data;
+				data.resize(data_size_a);
+				PoolByteArray::Write w = data.write();
+				print_line("Creating alpha image (shadowmask)");
+				memcpy(w.ptr(), output, data_size_a);
+				current_image_a->create(size.x, size.y, false, Image::FORMAT_L8, data);
+			}
+
+			// Add an alpha channel and merge the shadowmask back into the output texture.
+			denoised_image->convert(Image::FORMAT_RGBA8);
+			print_line(vformat("Denoised image size: %d x %d", denoised_image->get_width(), denoised_image->get_height()));
+			denoised_image->lock();
+			current_image_a->lock();
+			for (int y = 0; y < denoised_image->get_height(); y++) {
+				for (int x = 0; x < denoised_image->get_width(); x++) {
+					denoised_image->set_pixel(
+							x, y,
+							denoised_image->get_pixel(x, y) * Color(1, 1, 1, current_image_a->get_pixel(x, y).r));
+				}
+			}
+			current_image_a->unlock();
+			denoised_image->unlock();
 
 			PoolByteArray denoised_data = denoised_image->get_data();
 			denoised_image.unref();
 			PoolByteArray::Read r = denoised_data.read();
-			memcpy(output, r.ptr(), data_size);
+			// The data size is for RGBA8.
+			memcpy(output, r.ptr(), 4 * data_size_a);
 		}
 	}
 
@@ -1093,25 +1140,25 @@ void LightmapperCPU::_compute_seams(const MeshInstance &p_mesh, LocalVector<UVSe
 	}
 }
 
-void LightmapperCPU::_fix_seams(const LocalVector<UVSeam> &p_seams, Vector3 *r_lightmap, Vector2i p_size) {
-	LocalVector<Vector3> extra_buffer;
+void LightmapperCPU::_fix_seams(const LocalVector<UVSeam> &p_seams, Color *r_lightmap, Vector2i p_size) {
+	LocalVector<Color> extra_buffer;
 	extra_buffer.resize(p_size.x * p_size.y);
 
-	memcpy(extra_buffer.ptr(), r_lightmap, p_size.x * p_size.y * sizeof(Vector3));
+	memcpy(extra_buffer.ptr(), r_lightmap, p_size.x * p_size.y * sizeof(Color));
 
-	Vector3 *read_ptr = extra_buffer.ptr();
-	Vector3 *write_ptr = r_lightmap;
+	Color *read_ptr = extra_buffer.ptr();
+	Color *write_ptr = r_lightmap;
 
 	for (int i = 0; i < 5; i++) {
 		for (unsigned int j = 0; j < p_seams.size(); j++) {
 			_fix_seam(p_seams[j].edge0[0], p_seams[j].edge0[1], p_seams[j].edge1[0], p_seams[j].edge1[1], read_ptr, write_ptr, p_size);
 			_fix_seam(p_seams[j].edge1[0], p_seams[j].edge1[1], p_seams[j].edge0[0], p_seams[j].edge0[1], read_ptr, write_ptr, p_size);
 		}
-		memcpy(read_ptr, write_ptr, p_size.x * p_size.y * sizeof(Vector3));
+		memcpy(read_ptr, write_ptr, p_size.x * p_size.y * sizeof(Color));
 	}
 }
 
-void LightmapperCPU::_fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, const Vector2 &p_uv0, const Vector2 &p_uv1, const Vector3 *p_read_buffer, Vector3 *r_write_buffer, const Vector2i &p_size) {
+void LightmapperCPU::_fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, const Vector2 &p_uv0, const Vector2 &p_uv1, const Color *p_read_buffer, Color *r_write_buffer, const Vector2i &p_size) {
 	Vector2 line[2];
 	line[0] = p_pos0 * p_size;
 	line[1] = p_pos1 * p_size;
@@ -1160,8 +1207,8 @@ void LightmapperCPU::_fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, con
 		Vector2 current_uv = p_uv0 * (1.0 - t) + p_uv1 * t;
 		Vector2i sampled_point = (current_uv * p_size).floor();
 
-		Vector3 current_color = r_write_buffer[pixel.y * p_size.x + pixel.x];
-		Vector3 sampled_color = p_read_buffer[sampled_point.y * p_size.x + sampled_point.x];
+		Color current_color = r_write_buffer[pixel.y * p_size.x + pixel.x];
+		Color sampled_color = p_read_buffer[sampled_point.y * p_size.x + sampled_point.x];
 
 		r_write_buffer[pixel.y * p_size.x + pixel.x] = current_color * 0.6f + sampled_color * 0.4f;
 
@@ -1179,7 +1226,7 @@ void LightmapperCPU::_fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, con
 	}
 }
 
-void LightmapperCPU::_dilate_lightmap(Vector3 *r_lightmap, const LocalVector<int> p_indices, Vector2i p_size, int margin) {
+void LightmapperCPU::_dilate_lightmap(Color *r_lightmap, const LocalVector<int> p_indices, Vector2i p_size, int margin) {
 	for (int i = 0; i < p_size.y; i++) {
 		for (int j = 0; j < p_size.x; j++) {
 			int idx = p_indices[i * p_size.x + j];
@@ -1221,7 +1268,7 @@ void LightmapperCPU::_dilate_lightmap(Vector3 *r_lightmap, const LocalVector<int
 	}
 }
 
-void LightmapperCPU::_blit_lightmap(const Vector<Vector3> &p_src, const Vector2i &p_size, Ref<Image> &p_dst, int p_x, int p_y, bool p_with_padding) {
+void LightmapperCPU::_blit_lightmap(const Vector<Color> &p_src, const Vector2i &p_size, Ref<Image> &p_dst, int p_x, int p_y, bool p_with_padding) {
 	int padding = p_with_padding ? 1 : 0;
 	ERR_FAIL_COND(p_x < padding || p_y < padding);
 	ERR_FAIL_COND(p_x + p_size.x > p_dst->get_width() - padding);
@@ -1229,9 +1276,9 @@ void LightmapperCPU::_blit_lightmap(const Vector<Vector3> &p_src, const Vector2i
 
 	p_dst->lock();
 	for (int y = 0; y < p_size.y; y++) {
-		const Vector3 *__restrict src = p_src.ptr() + y * p_size.x;
+		const Color *__restrict src = p_src.ptr() + y * p_size.x;
 		for (int x = 0; x < p_size.x; x++) {
-			p_dst->set_pixel(p_x + x, p_y + y, Color(src->x, src->y, src->z));
+			p_dst->set_pixel(p_x + x, p_y + y, Color(src->r, src->g, src->b, src->a));
 			src++;
 		}
 	}
@@ -1241,16 +1288,16 @@ void LightmapperCPU::_blit_lightmap(const Vector<Vector3> &p_src, const Vector2i
 			int yy = CLAMP(y, 0, p_size.y - 1);
 			int idx_left = yy * p_size.x;
 			int idx_right = idx_left + p_size.x - 1;
-			p_dst->set_pixel(p_x - 1, p_y + y, Color(p_src[idx_left].x, p_src[idx_left].y, p_src[idx_left].z));
-			p_dst->set_pixel(p_x + p_size.x, p_y + y, Color(p_src[idx_right].x, p_src[idx_right].y, p_src[idx_right].z));
+			p_dst->set_pixel(p_x - 1, p_y + y, Color(p_src[idx_left].r, p_src[idx_left].g, p_src[idx_left].b, p_src[idx_left].a));
+			p_dst->set_pixel(p_x + p_size.x, p_y + y, Color(p_src[idx_right].r, p_src[idx_right].g, p_src[idx_right].b, p_src[idx_right].a));
 		}
 
 		for (int x = -1; x < p_size.x + 1; x++) {
 			int xx = CLAMP(x, 0, p_size.x - 1);
 			int idx_top = xx;
 			int idx_bot = idx_top + (p_size.y - 1) * p_size.x;
-			p_dst->set_pixel(p_x + x, p_y - 1, Color(p_src[idx_top].x, p_src[idx_top].y, p_src[idx_top].z));
-			p_dst->set_pixel(p_x + x, p_y + p_size.y, Color(p_src[idx_bot].x, p_src[idx_bot].y, p_src[idx_bot].z));
+			p_dst->set_pixel(p_x + x, p_y - 1, Color(p_src[idx_top].r, p_src[idx_top].g, p_src[idx_top].b, p_src[idx_top].a));
+			p_dst->set_pixel(p_x + x, p_y + p_size.y, Color(p_src[idx_bot].r, p_src[idx_bot].g, p_src[idx_bot].b, p_src[idx_bot].a));
 		}
 	}
 	p_dst->unlock();
@@ -1428,7 +1475,7 @@ LightmapperCPU::BakeError LightmapperCPU::bake(BakeQuality p_quality, bool p_use
 
 	raycaster.unref(); // Not needed anymore, free some memory.
 
-	LocalVector<LocalVector<Vector3>> lightmaps_data;
+	LocalVector<LocalVector<Color>> lightmaps_data;
 	lightmaps_data.resize(mesh_instances.size());
 
 	for (unsigned int i = 0; i < mesh_instances.size(); i++) {
@@ -1455,7 +1502,9 @@ LightmapperCPU::BakeError LightmapperCPU::bake(BakeQuality p_quality, bool p_use
 		for (int i = 0; i < atlas_slices; i++) {
 			Ref<Image> image;
 			image.instance();
-			image->create(atlas_size.x, atlas_size.y, false, Image::FORMAT_RGBH);
+			// FIXME: Baking alpha channel only works if image format is RGBA8, not RGBAH.
+			// However, using RGBA8 breaks HDR light baking, so only LDR light baking can be used for now.
+			image->create(atlas_size.x, atlas_size.y, false, Image::FORMAT_RGBA8);
 			bake_textures[i] = image;
 		}
 	} else {
@@ -1488,7 +1537,9 @@ LightmapperCPU::BakeError LightmapperCPU::bake(BakeQuality p_quality, bool p_use
 
 			Ref<Image> image;
 			image.instance();
-			image->create(mesh_instances[i].size.x, mesh_instances[i].size.y, false, Image::FORMAT_RGBH);
+			// FIXME: Baking alpha channel only works if image format is RGBA8, not RGBAH.
+			// However, using RGBA8 breaks HDR light baking, so only LDR light baking can be used for now.
+			image->create(mesh_instances[i].size.x, mesh_instances[i].size.y, false, Image::FORMAT_RGBA8);
 			image->set_name(mesh_name);
 			bake_textures[i] = image;
 		}
diff --git a/modules/lightmapper_cpu/lightmapper_cpu.h b/modules/lightmapper_cpu/lightmapper_cpu.h
index 7f64b3f8611b..a70f29dd47ff 100644
--- a/modules/lightmapper_cpu/lightmapper_cpu.h
+++ b/modules/lightmapper_cpu/lightmapper_cpu.h
@@ -75,6 +75,7 @@ class LightmapperCPU : public Lightmapper {
 
 		Vector3 direct_light;
 		Vector3 output_light;
+		float shadowmask = 1.0;
 
 		float area_coverage;
 	};
@@ -157,11 +158,11 @@ class LightmapperCPU : public Lightmapper {
 
 	void _post_process(uint32_t p_idx, void *r_output);
 	void _compute_seams(const MeshInstance &p_mesh, LocalVector<UVSeam> &r_seams);
-	void _fix_seams(const LocalVector<UVSeam> &p_seams, Vector3 *r_lightmap, Vector2i p_size);
-	void _fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, const Vector2 &p_uv0, const Vector2 &p_uv1, const Vector3 *p_read_buffer, Vector3 *r_write_buffer, const Vector2i &p_size);
-	void _dilate_lightmap(Vector3 *r_lightmap, const LocalVector<int> p_indices, Vector2i p_size, int margin);
+	void _fix_seams(const LocalVector<UVSeam> &p_seams, Color *r_lightmap, Vector2i p_size);
+	void _fix_seam(const Vector2 &p_pos0, const Vector2 &p_pos1, const Vector2 &p_uv0, const Vector2 &p_uv1, const Color *p_read_buffer, Color *r_write_buffer, const Vector2i &p_size);
+	void _dilate_lightmap(Color *r_lightmap, const LocalVector<int> p_indices, Vector2i p_size, int margin);
 
-	void _blit_lightmap(const Vector<Vector3> &p_src, const Vector2i &p_size, Ref<Image> &p_dst, int p_x, int p_y, bool p_with_padding);
+	void _blit_lightmap(const Vector<Color> &p_src, const Vector2i &p_size, Ref<Image> &p_dst, int p_x, int p_y, bool p_with_padding);
 
 public:
 	virtual void add_albedo_texture(Ref<Texture> p_texture);
diff --git a/scene/3d/baked_lightmap.cpp b/scene/3d/baked_lightmap.cpp
index e1f631c4be6e..ee24f5fa2070 100644
--- a/scene/3d/baked_lightmap.cpp
+++ b/scene/3d/baked_lightmap.cpp
@@ -546,11 +546,33 @@ void BakedLightmap::_save_image(String &r_base_path, Ref<Image> r_img, bool p_us
 	}
 	r_img->unlock();
 
-	if (!use_color) {
+	if (use_color) {
+		if (use_hdr) {
+			if (!use_shadowmask) {
+				// Discard the alpha channel which contains the shadowmask.
+				r_img->convert(Image::FORMAT_RGBH);
+			}
+		} else {
+			if (!use_shadowmask) {
+				// Discard the alpha channel which contains the shadowmask.
+				r_img->convert(Image::FORMAT_RGB8);
+			}
+		}
+	} else {
+		// Grayscale.
 		if (use_hdr) {
 			r_img->convert(Image::FORMAT_RH);
+			// Can't discard the alpha channel as there is no `Image::FORMAT_RAH`.
+			// There is `Image::FORMAT_RGH`, but it would require special handling in the shader.
 		} else {
-			r_img->convert(Image::FORMAT_L8);
+			if (use_shadowmask) {
+				// Discard color information to keep only luminance.
+				r_img->convert(Image::FORMAT_LA8);
+			} else {
+				// Discard color information to keep only luminance,
+				// and discard the alpha channel which contains the shadowmask.
+				r_img->convert(Image::FORMAT_L8);
+			}
 		}
 	}
 
@@ -740,13 +762,27 @@ BakedLightmap::BakeError BakedLightmap::bake(Node *p_from_node, String p_data_sa
 		lightmapper->add_mesh(md, lightmap_size);
 	}
 
+	// Used to print a warning when more than one baked DirectionalLight is present in the scene
+	// and shadowmasking is enabling (as this use case isn't supported).
+	String shadowmasked_light_name;
+
 	for (int i = 0; i < lights_found.size(); i++) {
 		Light *light = lights_found[i].light;
 		Transform xf = lights_found[i].xform;
 
 		if (Object::cast_to<DirectionalLight>(light)) {
 			DirectionalLight *l = Object::cast_to<DirectionalLight>(light);
+			if (shadowmasked_light_name != String()) {
+				// If shadowmasking is disabled, `shadowmasked_light_name` will remain an empty string,
+				// so this warning won't appear.
+				// If the second DirectionalLight has its bake mode set to Disabled,
+				// this part of the code will never be called either.
+				WARN_PRINT(vformat("The DirectionalLight \"%s\" is configured to cast shadows, but the DirectionalLight \"%s\" before it is already used for shadowmasking. This will lead to incorrect shadows in the distance.\nTo resolve this, set the DirectionalLight \"%s\"'s bake mode to Disabled or disable Use Shadowmask in the BakedLightmap node.", l->get_name(), shadowmasked_light_name, l->get_name()));
+			}
 			lightmapper->add_directional_light(light->get_bake_mode() == Light::BAKE_ALL, -xf.basis.get_axis(Vector3::AXIS_Z).normalized(), l->get_color(), l->get_param(Light::PARAM_ENERGY), l->get_param(Light::PARAM_INDIRECT_ENERGY), l->get_param(Light::PARAM_SIZE));
+			if (use_shadowmask) {
+				shadowmasked_light_name = l->get_name();
+			}
 		} else if (Object::cast_to<OmniLight>(light)) {
 			OmniLight *l = Object::cast_to<OmniLight>(light);
 			lightmapper->add_omni_light(light->get_bake_mode() == Light::BAKE_ALL, xf.origin, l->get_color(), l->get_param(Light::PARAM_ENERGY), l->get_param(Light::PARAM_INDIRECT_ENERGY), l->get_param(Light::PARAM_RANGE), l->get_param(Light::PARAM_ATTENUATION), l->get_param(Light::PARAM_SIZE));
@@ -1356,6 +1392,14 @@ bool BakedLightmap::is_using_color() const {
 	return use_color;
 }
 
+void BakedLightmap::set_use_shadowmask(bool p_enable) {
+	use_shadowmask = p_enable;
+}
+
+bool BakedLightmap::is_using_shadowmask() const {
+	return use_shadowmask;
+}
+
 void BakedLightmap::set_environment_mode(EnvironmentMode p_mode) {
 	environment_mode = p_mode;
 	_change_notify();
@@ -1493,15 +1537,18 @@ void BakedLightmap::_bind_methods() {
 	ClassDB::bind_method(D_METHOD("set_environment_min_light", "min_light"), &BakedLightmap::set_environment_min_light);
 	ClassDB::bind_method(D_METHOD("get_environment_min_light"), &BakedLightmap::get_environment_min_light);
 
-	ClassDB::bind_method(D_METHOD("set_use_denoiser", "use_denoiser"), &BakedLightmap::set_use_denoiser);
+	ClassDB::bind_method(D_METHOD("set_use_denoiser", "enable"), &BakedLightmap::set_use_denoiser);
 	ClassDB::bind_method(D_METHOD("is_using_denoiser"), &BakedLightmap::is_using_denoiser);
 
-	ClassDB::bind_method(D_METHOD("set_use_hdr", "use_denoiser"), &BakedLightmap::set_use_hdr);
+	ClassDB::bind_method(D_METHOD("set_use_hdr", "enable"), &BakedLightmap::set_use_hdr);
 	ClassDB::bind_method(D_METHOD("is_using_hdr"), &BakedLightmap::is_using_hdr);
 
-	ClassDB::bind_method(D_METHOD("set_use_color", "use_denoiser"), &BakedLightmap::set_use_color);
+	ClassDB::bind_method(D_METHOD("set_use_color", "enable"), &BakedLightmap::set_use_color);
 	ClassDB::bind_method(D_METHOD("is_using_color"), &BakedLightmap::is_using_color);
 
+	ClassDB::bind_method(D_METHOD("set_use_shadowmask", "use_denoiser"), &BakedLightmap::set_use_shadowmask);
+	ClassDB::bind_method(D_METHOD("is_using_shadowmask"), &BakedLightmap::is_using_shadowmask);
+
 	ClassDB::bind_method(D_METHOD("set_generate_atlas", "enabled"), &BakedLightmap::set_generate_atlas);
 	ClassDB::bind_method(D_METHOD("is_generate_atlas_enabled"), &BakedLightmap::is_generate_atlas_enabled);
 
@@ -1540,6 +1587,7 @@ void BakedLightmap::_bind_methods() {
 	ADD_PROPERTY(PropertyInfo(Variant::BOOL, "use_denoiser"), "set_use_denoiser", "is_using_denoiser");
 	ADD_PROPERTY(PropertyInfo(Variant::BOOL, "use_hdr"), "set_use_hdr", "is_using_hdr");
 	ADD_PROPERTY(PropertyInfo(Variant::BOOL, "use_color"), "set_use_color", "is_using_color");
+	ADD_PROPERTY(PropertyInfo(Variant::BOOL, "use_shadowmask"), "set_use_shadowmask", "is_using_shadowmask");
 	ADD_PROPERTY(PropertyInfo(Variant::REAL, "bias", PROPERTY_HINT_RANGE, "0.00001,0.1,0.00001,or_greater"), "set_bias", "get_bias");
 	ADD_PROPERTY(PropertyInfo(Variant::REAL, "default_texels_per_unit", PROPERTY_HINT_RANGE, "0.0,64.0,0.01,or_greater"), "set_default_texels_per_unit", "get_default_texels_per_unit");
 
@@ -1609,6 +1657,7 @@ BakedLightmap::BakedLightmap() {
 	use_denoiser = true;
 	use_hdr = true;
 	use_color = true;
+	use_shadowmask = true;
 	bias = 0.005;
 
 	generate_atlas = true;
diff --git a/scene/3d/baked_lightmap.h b/scene/3d/baked_lightmap.h
index 438c7a89d5c6..4a352c25b6d0 100644
--- a/scene/3d/baked_lightmap.h
+++ b/scene/3d/baked_lightmap.h
@@ -167,6 +167,7 @@ class BakedLightmap : public VisualInstance {
 	bool use_denoiser;
 	bool use_hdr;
 	bool use_color;
+	bool use_shadowmask;
 
 	EnvironmentMode environment_mode;
 	Ref<Sky> environment_custom_sky;
@@ -264,6 +265,9 @@ class BakedLightmap : public VisualInstance {
 	void set_use_color(bool p_enable);
 	bool is_using_color() const;
 
+	void set_use_shadowmask(bool p_enable);
+	bool is_using_shadowmask() const;
+
 	void set_bounces(int p_bounces);
 	int get_bounces() const;
 
diff --git a/scene/3d/light.cpp b/scene/3d/light.cpp
index 281a3f218a31..1acfadfd8208 100644
--- a/scene/3d/light.cpp
+++ b/scene/3d/light.cpp
@@ -198,7 +198,10 @@ void Light::_validate_property(PropertyInfo &property) const {
 		property.usage = PROPERTY_USAGE_NOEDITOR | PROPERTY_USAGE_INTERNAL;
 	}
 
-	if (bake_mode != BAKE_ALL && property.name == "light_size") {
+	// DirectionalLight can bake shadowmasks with its bake mode set to Indirect,
+	// so the light size is still relevant to display in this case.
+	const bool can_bake_shadowmask = type == VS::LIGHT_DIRECTIONAL && bake_mode == BAKE_INDIRECT;
+	if (!can_bake_shadowmask && bake_mode != BAKE_ALL && property.name == "light_size") {
 		property.usage = PROPERTY_USAGE_NOEDITOR | PROPERTY_USAGE_INTERNAL;
 	}
 }

Usage

This feature is enabled by default. It only supports 1 DirectionalLight at once. Point lights are outside the scope of shadowmasking, since the shadowmask is stored in the lightmap's alpha channel for now (and point lights' shadows don't disappear in the distance either).

If you don't want to store an alpha channel in the lightmap to decrease its file size, disable Use Shadowmask in the BakedLightmap's properties before baking. Bake times remain identical regardless of whether shadowmasking is enabled or not.

There are 3 ways to use it, depending on the target hardware:

• Default: Real-time shadow is used up close for both static and dynamic objects.
  • This does not provide a direct performance advantage, but makes static shadows never disappear in the distance and can make shadow acne less visible when up close (since the shadowmask is also sourced when up close). Nonetheless, it lets you decrease your DirectionalLight's Shadow Max Distance property, which increases the shadow detail up close and can improve performance.
  • You also get better shadow detail in the distance and less noticeable split transitions, depending on your lightmap's texel density.
  • This shadow mode is very similar to the one used by the most popular game on Steam as of writing 😉
• "Always Shadowmask": Real-time shadow is used up close for dynamic objects only. Static objects only source their shadow from the shadowmask.
  • This is achieved by selecting all the baked meshes and setting their cast shadows mode to Off after baking lightmaps. Eventually, a property may be added to do this in a more convenient way.
  • This is generally a good choice when targeting recent/high-end mobile platforms or integrated graphics.
• "Fully Static": Dynamic shadows do not cast shadows. Static objects only source their shadow from the shadowmask.
  • This is achieved by disabling the DirectionalLight shadow after baking lightmaps. Eventually, a property may be added to do this in a more convenient way.
  • Unlike using the All bake mode for the DirectionalLight, the DirectionalLight's color and shadow color can still be changed at run-time, without having to bake lightmaps again. This also allows using the same set of lightmaps to support real-time shadows for faster hardware.
  • This is generally a good choice when targeting old/low-end mobile platforms or integrated graphics.

Preview

No shadowmask (current)

without_shadowmask.mp4

With shadowmask (default for newly baked lightmaps)

with_shadowmask.mp4

TODO

• Port the shader changes to GLES2.
  • This is done and mostly working, but the shadow will fade suddenly at the end since it won't draw the shadowmask past the Shadow Max Distance distance unless the DirectionalLight has its shadow disabled. I don't know why.
• Restore support for baking lightmaps with HDR. See the FIXME in the diff for more information.
• Preserve the shadowmask when denoising is enabled. This likely involves moving the shadowmask to a different buffer, creating a new buffer that doesn't contain the shadowmask, passing that new buffer to the denoiser, then combining everything back.
• Handle cases where there is more than 1 DirectionalLight in the scene. We could print a warning message when baking lights in this case, and only take the first DirectionalLight into account for shadow masking.
• Blur the shadowmask slightly? Right now, the shadowmask is "all or nothing". It's fine at a distance when bicubic filtering is enabled, but it's not great when up close.
  • Done by allowing the adjustment of the DirectionalLight shadow size when its bake mode is Indirect (in addition to All). This property is used in shadowmask baking.
• Handle dynamic object capture. The issue is that we can't store the shadowmask into the probes' alpha channel, as the alpha channel is currently used to determine whether environment lighting should affect the object (it's a boolean choice). If we can move that usage outside of the probe color, then we can use the alpha channel for the shadowmask and use it to shade dynamic objects in the distance.
  • I don't think it's essential to do this in this PR; it can be done later if we don't figure out an easy way to do it.

[Calinou]

I've tried to get the denoiser working again, but I didn't succeed yet. Feel free to take a look at the diff and point out at anything that may be wrong 🙂

The goal is to preserve the final image's alpha channel. To do so, we split the RGBA image into a RGB image that is denoised by OpenImageDenoise and an alpha-only image that is kept as-is (stored as a L8 image internally). This is required as OpenImageDenoise does not support denoising images with an alpha channel – it will be stripped upon denoising.

Alternatively, we could modify the OpenImageDenoise integration to keep the alpha channel as-is in denoised images. This would make things easier if we need to denoise RGBA images elsewhere.

Denoiser disabled (working)

Denoiser enabled (broken)

[NHodgesVFX]

Not sure if you saw it but according to the docs you can make it so OIDN ignores extra channels like alpha.

"Images support only the OIDN_FORMAT_FLOAT3 pixel format. Custom image layouts with extra channels (e.g. alpha channel) or other data are supported as well by specifying a non-zero pixel stride. This way, expensive image layout conversion and copying can be avoided but the extra data will be ignored by the filter."

You can find the above text by searching for the word alpha.
https://www.openimagedenoise.org/documentation.html

[Calinou]

@NHodgesVFX Thanks a lot for the pointers!

I've tried to specify various stride sizes such as 4, 8, 12 and 16 for both the "color" and "output" images (including different strides for each). Unfortunately, I haven't gotten any combination that looks good – it always ends up looking corrupted somehow. I've pushed an updated version so you can take a look.

[punyrobot]

hey, so i was looking for a way to blend baked and realtime generated shadows, which seems a common ask, and which this pull request so nearly does...
i found by adding this line in scene.glsl i got pretty much what i wanted:
at line ~2151

#ifdef USE_LIGHTMAP
ambient_light = mix(vec3(1.0)(1.0-lightmap_sample.a),light_attenuation*lightmap_sample.a,lightmap_sample.a);
#endif

so attenuating ambient by the shadow map only if its not in the shadowmask

the workflow being bake a light map with shadowmask with direct and indirect for the static objects (with their meshes set to cast shadows)
then turn off cast shadows and set the light to indirect only or disabled

any thoughts?

(in this image the stadium shadows are baked and the players shadows are dynamically generated)

[mrjustaguy]

I have an idea how Alpha channel could be used to store both environment and Shadowmask data. since colors aren't clamped to 2 choices, you could have 4 different states, representing 00, 01, 10, 11 (one represents env, other shadowmask)
All that is needed is to pick 4 arbitrary values and assign which combination the values represent.
Example:
Alpha 0 = 00 (No Shadows, No Env Light)
Alpha 0.3 = 01 (No Shadows, Env Light)
Alpha 0.6 = 10 (Shadows, No Env Light)
Alpha 1 = 11 (Shadows, Env Light)

[Calinou]

Rebased and tested again (including support for asynchronous shader compilation in GLES3).

[mrjustaguy]

I have been unable to get Shadow masks to work with the artifact created today.
Lightmap.zip

Nevermind, Had to change Directional light Bake mode from all.

[Calinou]

@NHodgesVFX Please don't bump issues (or pull requests) without contributing significant new information. Use the 👍 reaction button on the first post instead.

[Calinou]

• Superseded by [3.x] Implement shadowmask for DirectionalLight in BakedLightmap #68702 (3.x) and LightmapGI: Implement shadowmask for DirectionalLight #77284 Implement LightmapGI shadowmasks #85653 (4.x). Thanks @DearthDev for continuing this work 🙂