Pathfinding is going off the navmesh

[emrys90]

I have a navmesh that looks like this

My pathfinding is causing the agent to walk through the water, which is the empty area in the center of the screenshot.

Any ideas what I am doing wrong? Here's my code for finding the path. I probably screwed something up, it wasn't easy trying to figure out how to actually find a path using DotRecast. Had to use OpenAI and a lot of trial and error to try and figure it out lol.

    private unsafe void FindPath(DtNavMesh navMesh, PathRequest request)
    {
        var startPosition = new RcVec3f(request.StartPosition.x, request.StartPosition.y, request.StartPosition.z);
        var endPosition = new RcVec3f(request.EndPosition.x, request.EndPosition.y, request.EndPosition.z);
        var navQuery = new DtNavMeshQuery(navMesh);

        // Find the start and end polygons on the navmesh
        var filter = DtQueryNoOpFilter.Shared;
        var halfExtents = new RcVec3f(2.0f, 4.0f, 2.0f); // Extent for search area

        var startPolyResult = navQuery.FindNearestPoly(startPosition, halfExtents, filter, out var startPolyRef, out _, out _); 
        if (!startPolyResult.Succeeded())
        {
            this.LogError($"Failed to find start position reference: {startPolyResult}");
            request.Failed();
            return;
        }

        var endPolyResult = navQuery.FindNearestPoly(endPosition, halfExtents, filter, out var endPolyRef, out _, out var canReachTarget);
        if (!endPolyResult.Succeeded())
        {
            this.LogError($"Failed to find end position reference: {endPolyResult}");
            request.Failed();
            return;
        }
        
        _pathCache.Clear();
        var pathResult = navQuery.FindPath(startPolyRef,
            endPolyRef,
            startPosition,
            endPosition,
            filter,
            ref _pathCache,
            DtFindPathOption.AnyAngle
        );

        if (!pathResult.Succeeded())
        {
            this.LogError("Failed to find end position reference");
            request.Failed();
            return;
        }

        // Output the path as an array of points
        Span<DtStraightPath> straightPath = stackalloc DtStraightPath[256];
        navQuery.FindStraightPath(startPosition, endPosition, _pathCache, _pathCache.Count, straightPath, out var straightPathCount, 256, 0);
        
        var resultPath = new NetVector3[straightPathCount];
        for (int i = 0; i < straightPathCount; i++)
        {
            var pos = straightPath[i].pos;
            resultPath[i] = new NetVector3(pos.X, pos.Y, pos.Z);
        }

        request.Success(resultPath, canReachTarget);
    }

[emrys90]

Here's how I'm loading the NavMesh

    private async void LoadNavMesh()
    {
        try
        {
            await using var fileStream = new FileStream("Actors/World/World.bytes", FileMode.Open);
            using var br = new BinaryReader(fileStream);
            var reader = new DtMeshSetReader();
            var navMesh = reader.Read(br, 6);
            
            ProcessQueue(navMesh);
        }
        catch (Exception ex)
        {
            this.LogException(ex);
        }
    }

And here's how I'm generating the NavMesh, by converting Unity's NavMesh to Recast.

        private async UniTask PerformBake()
        {
            this.Log("Enabling baked colliders");
            foreach (var bakedCollider in _bakedColliders)
            {
                bakedCollider.enabled = true;
            }
            
            this.Log("Baking Unity NavMesh");
            var targets = new Object[] { _navMesh };
            NavMeshAssetManager.instance.StartBakingSurfaces(targets);
            await UniTask.WaitUntil(() => !NavMeshAssetManager.instance.IsSurfaceBaking(_navMesh));
            
            this.Log("Disabling baked colliders");
            foreach (var bakedCollider in _bakedColliders)
            {
                bakedCollider.enabled = false;
            }
            
            NavMeshTriangulation triangles = NavMesh.CalculateTriangulation();
            Mesh mesh = new Mesh();
            mesh.vertices = triangles.vertices;
            mesh.triangles = triangles.indices;
            
            var rc = new UniRcNavMeshSurfaceTarget("World", mesh, Matrix4x4.identity);
            var rcMesh = rc.ToMesh();
            rcMesh.SaveFile();

            var settings = new RcNavMeshBuildSettings()
            {
                //cellSize = 0.3f,
                //cellHeight = 0.2f,

                agentHeight = 2.0f,
                agentRadius = 0f,
                agentMaxClimb = 0.9f,
                agentMaxSlope = 45f,

                agentMaxAcceleration = 8.0f,
                agentMaxSpeed = 3.5f,

                minRegionSize = 0,
                mergedRegionSize = 20,

                //partitioning = RcPartitionType.WATERSHED.Value,

                filterLowHangingObstacles = true,
                filterLedgeSpans = true,
                filterWalkableLowHeightSpans = true,

                edgeMaxLen = 12f,
                edgeMaxError = 1.3f,
                vertsPerPoly = 6,

                detailSampleDist = 6f,
                detailSampleMaxError = 1f,

                tiled = false,
                tileSize = _navMesh.tileSize,

                keepInterResults = true, // full memory
                buildAll = true,
            };
            var navMesh = rcMesh.Build(settings);
            navMesh.SaveNavMeshFile(rc.GetName());
            
            this.Log("Clearing Unity NavMesh data");
            NavMeshAssetManager.instance.ClearSurfaces(targets);
        }

[ikpil]

Hello. Nice to meet you! If it's okay, could I please get the navmesh file?

Oh, and if you have a screenshot of the reproduction steps for the issue, that would be really helpful. I can't quite figure out what the problem is from the screenshots you provided.

[emrys90]

Sure, thanks for helping with this!
World.zip

I'm trying to prevent the AI from walking into the water when following the player.

The pathfinding will even follow the player into the air. The path returned is just two waypoints, the first being the AI's current location and the end result being the target location. So its as if its not even trying to find the path and just going directly to the result.

[ikpil]

When I checked what you provided, I'm not sure what the problem is.

Based on the code you provided:

1. You need to check whether the values of startPolyRef and endPolyRef are less than or equal to 0.
2. You can learn how to use RctestNavMeshTool::FindFollowPath by looking at its usage."

[emrys90]

@ikpil Thanks for the suggestions!

1. Both are above 0, the values are 281475431792640 for the start, and 281475431792640 for the end. This is for a navigation path when the player is standing in the water area.

2. I can't find any differences between RctestNavMeshTool::FindFollowPath and my code, it appears to be similar in how the path is called. I thought maybe it was a coordinate system issue with Recast being different from Unity, but when I converted the coordinates, that caused the FindNearestPoly calls to fail.

[emrys90]

Progress! I found out its a coordinate conversion issue. What's weird is to fix it I had to do a negative X value for converting from Unity to Recast, but what I found online is its supposed to be a negative Z, so that's a bit odd and no idea why its different.

I've got 2 remaining issues I haven't been able to figure out yet though.

1. The bridge here, the pathfinding isn't raising up in the air. It instead stays at the same height of the terrain the bridge connects to. This happens if the target position is below the navmesh, so its like its taking the height from the target position instead of the navmesh height.
   

2. The pathfinding doesn't find the closest path if the target is off the navmesh, its like it gives up. I tried calling DtNavMeshQuery.ClosestPointOnPoly, but it didn't seem to make a difference.
   

[emrys90]

Any ideas on those two issues? I haven't been able to figure them out

[ikpil]

2. In most cases, the endpoint is determined using the FindNearestPoly function and its out nearestPt parameter.

[ikpil]

As for question 1, I'm not entirely sure what case you are referring to. When I opened the demo, I could see the path. Are you saying that the path goes under the bridge?

[emrys90]

As for question 1, I'm not entirely sure what case you are referring to. When I opened the demo, I could see the path. Are you saying that the path goes under the bridge?

Correct, the path returned does not have elevation in it, it just returns the same elevation the path started at.

[ikpil]

[ikpil]

[ikpil]

If you need to express climbing the stairs more closely, referring to the Pathfind Follow logic will be helpful. It is designed to return a smooth point at every 0.5 distance.

[ikpil]

world.2024-12-10.235932.mp4

[emrys90]

Okay I think it might be related to the NavMesh conversion. Here's how Unity's NavMesh looks like after baking.

And here's the NavMesh converted to DotRecast.

The Unity one is a lot smoother, and correctly matches the elevation of the bridge, while the DotRecast one appears like it would clip inside the bridge.

Any ideas how to fix this? Here's my conversion code.

        private async UniTask PerformBake()
        {
            this.Log("Baking Unity NavMesh");
            var targets = new Object[] { _navMesh };
            NavMeshAssetManager.instance.StartBakingSurfaces(targets);
            await UniTask.WaitUntil(() => !NavMeshAssetManager.instance.IsSurfaceBaking(_navMesh));
            
            NavMeshTriangulation triangles = NavMesh.CalculateTriangulation();
            Mesh mesh = new Mesh();
            mesh.vertices = triangles.vertices;
            mesh.triangles = triangles.indices;
            
            var rc = new UniRcNavMeshSurfaceTarget("World", mesh, Matrix4x4.identity);
            var rcMesh = rc.ToMesh();
            rcMesh.SaveFile();

            var settings = new RcNavMeshBuildSettings()
            {
                //cellSize = 0.3f,
                //cellHeight = 0.2f,

                agentHeight = 2.0f,
                agentRadius = 0f,
                agentMaxClimb = 0.9f,
                agentMaxSlope = 45f,

                agentMaxAcceleration = 8.0f,
                agentMaxSpeed = 3.5f,

                minRegionSize = 0,
                mergedRegionSize = 20,

                //partitioning = RcPartitionType.WATERSHED.Value,

                filterLowHangingObstacles = true,
                filterLedgeSpans = true,
                filterWalkableLowHeightSpans = true,

                edgeMaxLen = 12f,
                edgeMaxError = 1.3f,
                vertsPerPoly = 6,

                detailSampleDist = 6f,
                detailSampleMaxError = 1f,

                tiled = false,
                tileSize = _navMesh.tileSize,

                keepInterResults = true, // full memory
                buildAll = true,
            };
            var navMesh = rcMesh.Build(settings);
            navMesh.SaveNavMeshFile(rc.GetName());
            
            this.Log("Clearing Unity NavMesh data");
            NavMeshAssetManager.instance.ClearSurfaces(targets);
        }

[ikpil]

First, NavMesh.CalculateTriangulation(); returns a simple triangle value for the Unity NavMesh. This means that there is no detailed information for walking paths. So, if you create a NavMesh for DotRecast based on this value, you will inevitably get a distorted shape.

To use DotRecast in Unity, you need to correctly create and input terrain data. You can refer to UniRecast's UniRcNavMeshSurface.BakeFrom for assistance.

Reference:

• https://docs.unity3d.com/ScriptReference/AI.NavMesh.CalculateTriangulation.html

[ikpil]

Or could the following code be helpful? I haven’t tried it myself, so it might not be accurate.

• Can I use Unity's navigation workflow to generate navmesh? UniRecast#6

[emrys90]

Thanks for the information, that's unfortunate about NavMesh.CalculateTriangulation.

I tried using UniRcNavMeshSurface, and here's the results. It doesn't seem to be baking well. Also I can't figure out how to remove the water area from the baked mesh like I do in Unity using a NavMesh Modifier Volume.

Compared to how it should look when baked with Unity.

[ikpil]

In the sample, it works fine. Could it be a bug?
If it is a bug, what should I focus on to fix it?

[emrys90]

In the sample, it appears to be using only one mesh. Have you tried it in a scene with a terrain and many meshes on the terrain?