Issue about compiling Ginkgo with rocm 6.0

[tpadioleau]

I recently had trouble compiling Ginkgo 1.6 with rocm 6.0. At first I had an issue with cmake that could not find a function called hip_add_library. After setting CMAKE_MODULE_PATH, I encountered /opt/rocm/hip/../llvm/bin/clang: not found because the directory hip does not exist in /opt/rocm. After creating this directory, I encountered

/ginkgo/hip/base/types.hip.hpp:46:10: fatal error: 'hipblas.h' file not found
#include <hipblas.h>

Are you aware of such problems ?

I attach two docker files, one that builds fine with rocm 5.7.3 (Dockerfile-rocm5.7.3.txt) and the one failing with rocm 6.0 (Dockerfile-rocm6.0.txt). Please let me know if you find something wrong in these files.

[tcojean]

Thanks a lot for the report and the details! I believe this is normal, we have not yet tested ROCm 6.0 and thankfully, ROCm 6.0 now enforces the FHS standard within its directory. This is generally good news as many things can be simplified, but we have not proofed our CMake build system for that yet.

Alternatively, maybe we can finally adopt the native HIP language support within CMake, and rewrite (simplify) our CMake HIP build system, which should also sidestep the issue.

[tpadioleau]

Ok thanks the quick reply!

[upsj]

I can confirm that #1334 works on ROCm 6.0.2

[lahwaacz]

I managed to build ginkgo 1.7.0 with ROCm 6.0 by setting the following env vars. That is, without benchmarks which are affected by #1566

export ROCM_PATH=/opt/rocm
export HIP_PATH="$ROCM_PATH"
export HIP_THRUST_PATH="$HIP_PATH"/include

[upsj]

Should be fixed by #1334

[lahwaacz]

@upsj hip_path.cmake still defaults to /opt/rocm/hip for HIP_PATH which does not work with ROCm 6.0:

ginkgo/cmake/hip_path.cmake

Line 3 in a8a407f

set(HIP_PATH "/opt/rocm/hip" CACHE PATH "Path to which HIP has been installed")

Furthermore, hip.cmake has a code path where $ENV{HIP_PATH}/.. is used for `ROCM_PATH:

ginkgo/cmake/hip.cmake

Line 20 in a8a407f

set(ROCM_PATH "$ENV{HIP_PATH}/.." CACHE PATH "Path to which ROCM has been installed")

It would be nice if Ginkgo could auto-detect ROCm version and use the correct paths for 6.0 by default 😉

[lahwaacz]

Actually, do you need any special handling for the paths at all? CMake can auto-detect ROCm/HIP just fine...

[upsj]

The only reason we need this right now is to enable autodetection of the hipBLAS/... paths, but that should be handled correctly by an environment where they are all part of the CMAKE_PREFIX_PATH. So hopefully we don't need it at all.