lidar_centerpoint does not compile with CUDA 12.5

[PPeltzerFka]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

Building Universe fails for me when following the Source installation guide, with running the ./setup-dev-env.sh on Ubuntu 22, installing the latest Cuda 12.5 (released in May). When trying to build Autoware Universe, the compilation fails for lidar_centerpoint, pointing to missing declaration for the __syncthreads() function and more.

Expected behavior

Following the tutorial, compilation is expected to be successful.

Actual behavior

The compilation fails for lidar_centerpoint. Unfortunately, I only have a part of the error message right now as I dont have the time capacity to reinstall CUDA 12.5 again. This one was originating from test_preprocess_kernel.cpp :

/usr/local/cuda/include/cub/util_ptx.cuh:271:5: error: ‘__syncthreads’ was not declared in this scope
  271 |     __syncthreads();

Steps to reproduce

1. Run ./setup-dev-env.sh
2. Validate you're running CUDA 12.5 (nvcc --version)
3. Build autoware universe, or only up to lidar_centerpoint: colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-up-to autoware_lidar_centerpoint

Versions

• OS: Ubuntu 22
• ROS 2: Humble
• Autoware: Initially tried with 2024.06 tag, but issue appears to be still there in latest Universe main commit

Possible causes

Google spits out the following threads related to this:

• Unable to include thrust/host_vector.h and others with CUDA 12.5
• When upgrade from CUDA12.4 to 12.5 the compilation became broken

There appears to be an API change in the thrust library released in CUDA 12.5, no longer allowing the inclusion of certain (?) headers in the .cpp/.hpp files instead of .cu files (judging from the linked topics).

lidar_centerpoint includes thrust libraries in a couple of files:

test_preprocess_kernel.cpp

#include <thrust/host_vector.h>
#include <thrust/reduce.h>

circle_nms_kernel.hpp

#include <thrust/device_vector.h>

postprocess_kernel.hpp

#include <thrust/device_vector.h>

test_preprocess_kernel.cpp

#include <thrust/host_vector.h>
#include <thrust/reduce.h>

Note that the lidar_transfusion package has the same include pattern, so the issue is probably present there as well.

Additional context

When removing CUDA 12.5 and installing CUDA 12.4, compilation is successful.

[amadeuszsz]

Hi @PPeltzerFka, thank you for your detailed report.

Currently Autoware is developed with CUDA 12.3. I can reproduce your issue and seems CUDA 12.5 came with API breaking changes. For the time being, could you use CUDA 12.3 / 12.4? Also here is another link which cover this issue.

EDIT:
Workaround will be setting appropriate device system before affected header e.g.

#define THRUST_DEVICE_SYSTEM THRUST_DEVICE_SYSTEM_CPP

I didn't test it yet but I guess it will bring performance drawbacks. I think we can just use headers in our CUDA library target directly. Could you please test this branch if it solves your issue?

[amadeuszsz]

Hi @PPeltzerFka
We made a fix PRs. I will appreciate if you could let us know later you can build and run Autoware successfully on your machine 🙇🏻‍♂️