HIPify'd code not finding NVIDIA GPU

[ArcherSmythe]

When trying to run accelerated code, I'm getting this error message:
CHIP critical [TID 5040] [1694286011.517175643] : Selected OpenCL device 0 is out of range
The error occurs when I try to run my binary or when trying to run any of ChipStar's C++ samples.

Running clinfo does show the GPU I have in the system:

  Platform Name                                   NVIDIA CUDA
Number of devices                                 1
  Device Name                                     Quadro K620

Using ldd on my binary indicates that /usr/lib64/libOpenCL.so is being used. I tried setting LD_LIBRARY_PATH to point to NVIDIA's library directory, but ldd still shows that my binary will use /usr/lib64/libOpenCL.so. I have CUDA 10.1 and 12.2 installed, so I tried replacing /usr/lib64/libOpenCL.so.1.0.0 with a symlink to NVIDIA's library. 10.1 didn't support OpenCL 2.1, so that failed. Symlinking to 12.2's libOpenCL yields the same "out of range" error.

If it helps, these are the commands I used to build my binary.

$ export CHIP_BE=opencl
$ export HIP_PLATFORM=spirv
$ CMAKE_PREFIX_PATH=/disk0/vendor/chipStar/2023-09-01/lib/cmake/hip CC=/disk0/vendor/llvm/16/bin/clang CXX=/disk0/vendor/llvm/16/bin/clang++ cmake ../../sbx/cuda/SampleCMake/
-- The CXX compiler identification is Clang 16.0.5
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /disk0/vendor/llvm/16/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- hip-config.cmake chipStar:
-- HIP_PATH:
-- HIP_COMPILER: clang
-- HIP_RUNTIME: spirv
-- HIP_PLATFORM: spirv
-- HIP_ARCH: spirv
-- HIP_OFFLOAD_COMPILE_OPTIONS:
-- HIP_OFFLOAD_LINK_OPTIONS:
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components: doxygen missing components: dot
-- Configuring done (1.6s)
-- Generating done (0.2s)
-- Build files have been written to: build/hipped

Setting CHIP_LOGLEVEL=debug doesn't give any extra output.
Setting CHIP_DEVICE_TYPE=all doesn't help. Setting it to CHIP_DEVICE_TYPE=foo results in the expected InvalidDeviceType error.

Secondary question: The section Compiling CUDA application directly with chipStar in Using.md: does that mean I should be able to compile my existing CUDA code without needing to run hipify? (I'd still need to update CMake files and possibly rename .cu files to .cpp.)

[pjaaskel]

I think NVIDIA's driver doesn't (yet) support SPIR-V which is a hard requirement for chipStar.
For the secondary question, yes, this is the aim. Direct CUDA compilation is a still heavily WiP though.

[ArcherSmythe]

Is there another way to compile HIP to OpenCL? I can't tell what AMD's ROCm actually does. Or maybe my issue is that I should be compiling the HIP to CUDA when I need to use NVIDIA, and HIP to SPIR-V on other platforms.

[pjaaskel]

Yes, I think that's the current best way. Perhaps this will change in the future towards being able to use chipStar as the portable toolchain for CUDA/HIP that produces also portable fat binaries (using SPIR-V).

[pjaaskel]

You can try PoCL-CUDA (experimental): pocl/pocl#1301 but beware of bumps and do not hesitate to contribute ;-)