Tagging objects for semantic segmentation suppresses render occlusion

[nsubiron]

Currently we tag objects by setting their custom depth stencil value, this way it automatically works for semantic segmentation as well. It turns out that setting this value makes objects being rendered even if they are behind another object. This occurs even if the custom depth pass is disabled at project level.

[nsubiron]

For now we disable custom depth pass for each object if semantic segmentation is not needed. However, it is still active in the project settings by default.

If necessary, can be disabled (in the release binary) by adding the file "CarlaUE4/Saved/Config/LinuxNoEditor/Engine.ini" containing the following

[/Script/Engine.RendererSettings]
r.CustomDepth=1

[juaxix]

This also makes to make an extra pass for the custom depth render, thus is sucking performance. The solution EPIC proposes is to make a material property to be applied in all the materials of all the objects in the world and then set the values for the stencil with the id of the object type for semantic segmentation, that will be used later in the post-fx shader...not an easy way to do it though

[germanros1987]

I guess there are no update for this issue, right?

[juaxix]

@germanros1987 not yet, this issue is a particular hard one, considering the other option is to change the core of the engine instead of doing the material trick, we are waiting for another solution from the Epic team they are digging into the problem btw.

[germanros1987]

@bernatx Is this still an active problem?

[bernatx]

Hi, I don't detect that problem anymore. As you can see in these screenshots, All objects show correctly in depth with occlusion.

[bernatx]

[nsubiron]

The problem cannot be seen in the cameras, the thing is that UE4 stops doing occlusion culling for meshes with the stencil buffer enabled (used to encode semseg). The resulting image is the same, but you're rendering more meshes than necessary, so it's just a performance issue. I have no idea if it's still happening but I suppose it is, since the main use for the stencil buffer is precisely to render occluded objects, probably the only solution is to use a different mechanism to render semantic segmentation.

[RachithP]

@juaxix, #4 (comment)

@germanros1987 not yet, this issue is a particular hard one, considering the other option is to change the core of the engine instead of doing the material trick, we are waiting for another solution from the Epic team they are digging into the problem btw.

It would be helpful if there is a tracking/bug number from Epic for reference. Thanks!

[germanros1987]

@RachithP I believe @juaxix was overexcited... EpicGames is NOT working on this.

[juaxix]

@RachithP I believe @juaxix was overexcited... EpicGames is NOT working on this.

Of course I was overexcited and still, we need to push harder on this, other companies in UDN are requesting the same thing, to control whether or not it is subject to Occlusion Culling on an actor-by-actor basis, not limited to Custom Depth.
But anyway, you can try this https://answers.unrealengine.com/questions/54283/render-occluded-actor-in-custom-depth.html meanwhile . The code for this part remains the same for 4.26 of course

bool FStaticMeshSceneProxy::CanBeOccluded() const
{
	return !MaterialRelevance.bDisableDepthTest && !ShouldRenderCustomDepth();
}