Impossible to render smooth cone mesh properly after #5666

[HackerFoo]

When rendering a smooth cone mesh, there is no way to assign a normal to the tip vertex of each triangle on the side without creases. Fortunately, you can assign the normal to zero, and the math works out if normals are normalized in the fragment shader.

See here for details: https://stackoverflow.com/questions/15283508/low-polygon-cone-smooth-shading-at-the-tip/37187730#37187730

#5666 normalizes in the vertex shader instead, which breaks this.

With #5666:

After reverting #5666:

[HackerFoo]

This happens because normalizing zero produces an invalid result.

Normalizing in the vertex shader invalidates all of the triangles but the bottom, but when normalizing in the fragment shader, only the pixel at the tip is invalid; otherwise the normals on the side are properly interpolated before normalization.

[HackerFoo]

Also, note that if normals may not be normalized in the fragment shader despite being normalized in the vertex shader, because interpolating normalized vectors does not necessarily result in a normalized vector. Larger angles between interpolated normals will result in smaller magnitude normals, so this won't be apparent with dense or flat meshes.

[alice-i-cecile]

CC @superdump.

[superdump]

At first glance, this seems like a hack. I appreciate that it works, but I wonder if there is another way.

I would rather step back and try to tackle the original problem than this solution with a vertex normal of 0,0,0. What is your mesh? A triangle list of triangles arranged to form a cone? If using a normal that points out of the tip, what do you see?

[superdump]

Also, note that if normals may not be normalized in the fragment shader despite being normalized in the vertex shader, because interpolating normalized vectors does not necessarily result in a normalized vector. Larger angles between interpolated normals will result in smaller magnitude normals, so this won't be apparent with dense or flat meshes.

If there is no other way then it could be that when using normal mapping we normalise in the vertex shader as mikktspace specified that this is how it must be done to be an exact inverse of the baking process as it was designed to be cheaper to calculate if possible and so the baked normal maps normalise the vertex normal at the fragment stage, and then when not normal mapping, we normalise in the fragment shader.

[HackerFoo]

At first glance, this seems like a hack. I appreciate that it works, but I wonder if there is another way.

It seems like a hack, but it's actually a reasonable solution, explained below.

I would rather step back and try to tackle the original problem than this solution with a vertex normal of 0,0,0. What is your mesh? A triangle list of triangles arranged to form a cone? If using a normal that points out of the tip, what do you see?

A cone is a (tessellated) circular base, surrounded by triangles that all meet at the tip, represented as a triangle list.

The normals for all points but the tip are easy to calculate. There are two obvious candidates for the tip normal:

1. The average of the base normals - this produces ugly creases, because each triangle flattens as the average normal takes over towards the tip.

Example:

Notice the base is smooth, with increasingly sharp creases to the tip.

Zero normal tip (correct):

2. In the direction of the tip - this is completely wrong, shading the cone as if the tip was round. Imagine an infinitely long cone, and the surrounding interpolated normals will be almost perpendicular to their true normal.

Example (looking down the tip):

Notice that, except for shadows, it looks like a ball.

Zero normal tip (correct):

What is needed is to blend the normals from the base vertices, but all the way to the tip. This is exactly what a zero tip normal does, because if you look at the three vertices normals as weighted normals, then the tip contributes nothing, as if each point on the triangle surface was projected to the base of the cone before calculating the normal. This the same smooth shading throughout the triangle.

[HackerFoo]

Summarized from discussion:

Using a loop somewhere between the tip and the base won't work, because it will break the smooth surface.
Using a loop at the tip (a cylinder with one end shrunk to a point) won't work. This would replace each triangle with a quad, but when the quads are split into triangles, you end up with two triangles for each quad, where one is degenerate (a line, because two of its vertices are at the tip.) The remaining triangles suffer from the same problem as (1) above.