Eliminate deprecated find_package(CUDA) from qmcpack

[quantumsteve]

Please review the developer documentation
on the wiki of this project that contains help and requirements.

Proposed changes

Describe what this PR changes and why. If it closes an issue, link to it here
with a supported keyword.

This replaces find_package(CUDA) with first-class language support and find_package(CUDAToolkit).

On Ubuntu 20.04 and CUDA HPC SDK 21.7, I have to manually set the math libraries and headers. This appears to be fixed in CMake master https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6405

On sulfur, I am encountering ABI issues between the old copy-on-write and small string optimized std::string. I assume it's a configuration issue on my part, but haven't tracked down where this is coming from and would like to run the tests here to confirm.

What documentation needs to be changed?

Fixes #3460

What type(s) of changes does this code introduce?

Delete the items that do not apply

• Build related changes

Does this introduce a breaking change?

• No

What systems has this change been tested on?

Checklist

Update the following with a yes where the items apply. If you're unsure about any of them, don't hesitate to ask. This is
simply a reminder of what we are going to look for before merging your code.

• Yes. This PR is up to date with current the current state of 'develop'
• Yes/No. Documentation has been added (if appropriate)

[qmc-robot]

Can one of the admins verify this patch?

[ye-luo]

Test this please

[ye-luo]

Test this please

[ye-luo]

Test this please

[ye-luo]

Test this please

[ye-luo]

In the case of NVHPC, I used CUDAToolkit_ROOT pointing to an standalone installation.

[quantumsteve]

In the case of NVHPC, I used CUDAToolkit_ROOT pointing to an standalone installation.

That worked well for CMake configuration/generation, but get errors about missing cublas headers.

On my machine (Ubuntu 20.04, NVHPC 21.7) I had to add `"${CUDAToolkit_INCLUDE_DIRS}/../../../../../math_libs/include" as an additional include directory.

[prckent]

Please verify that this works with Summit's modules, software configuration etc.

[ye-luo]

@quantumsteve could you try again the current branch and paste what "errors about missing cublas headers." are. I'm afraid of missing/incorrect dependencies.

[ye-luo]

@prckent update to Summit build recipes has been included in the last commit.

[quantumsteve]

@quantumsteve could you try again the current branch and paste what "errors about missing cublas headers." are. I'm afraid of missing/incorrect dependencies.

[125/663] Building CXX object src/QMCWaveFunctions/CMakeFiles/qmcwfs.dir/ElectronGas/ElectronGasOrbitalBuilder.cpp.o
FAILED: src/QMCWaveFunctions/CMakeFiles/qmcwfs.dir/ElectronGas/ElectronGasOrbitalBuilder.cpp.o 
/usr/bin/c++ -DADD_ -DH5_USE_16_API -DHAVE_CONFIG_H -DHAVE_EINSPLINE -DHAVE_LIBBOOST -DHAVE_LIBFFTW -DHAVE_LIBHDF5 -DOPENMP_NO_COMPLEX -DUSE_REAL_STRUCT_FACTOR -Drestrict=__restrict__ -I/home/svh/Documents/qmcpack/build/src -I/home/svh/Documents/qmcpack/src -I/home/svh/Documents/qmcpack/external_codes/boost_multi -I/home/svh/Documents/qmcpack/src/QMCWaveFunctions -I/home/svh/Documents/qmcpack/src/Particle -I/home/svh/Documents/qmcpack/src/Numerics -I/home/svh/Documents/qmcpack/src/io -I/home/svh/Documents/qmcpack/src/Containers -I/home/svh/Documents/qmcpack/src/Platforms -I/home/svh/Documents/qmcpack/src/Utilities -isystem /opt/nvidia/hpc_sdk/Linux_x86_64/21.7/cuda/11.4/include -isystem /usr/include/hdf5/serial -isystem /usr/include/libxml2 -fopenmp -finline-limit=1000 -fstrict-aliasing -funroll-all-loops -Wno-deprecated -Wvla -Wcomment -Wmisleading-indentation -Wmaybe-uninitialized -Wuninitialized -Wreorder -Wno-unknown-pragmas -Wno-sign-compare -Wsuggest-override -ffast-math -march=native -O3 -DNDEBUG -std=c++17 -MD -MT src/QMCWaveFunctions/CMakeFiles/qmcwfs.dir/ElectronGas/ElectronGasOrbitalBuilder.cpp.o -MF src/QMCWaveFunctions/CMakeFiles/qmcwfs.dir/ElectronGas/ElectronGasOrbitalBuilder.cpp.o.d -o src/QMCWaveFunctions/CMakeFiles/qmcwfs.dir/ElectronGas/ElectronGasOrbitalBuilder.cpp.o -c /home/svh/Documents/qmcpack/src/QMCWaveFunctions/ElectronGas/ElectronGasOrbitalBuilder.cpp
In file included from /home/svh/Documents/qmcpack/src/QMCWaveFunctions/Fermion/DelayedUpdateCUDA.h:19,
                 from /home/svh/Documents/qmcpack/src/QMCWaveFunctions/Fermion/DiracDeterminant.h:28,
                 from /home/svh/Documents/qmcpack/src/QMCWaveFunctions/Fermion/DiracDeterminantWithBackflow.h:24,
                 from /home/svh/Documents/qmcpack/src/QMCWaveFunctions/Fermion/SlaterDetWithBackflow.h:18,
                 from /home/svh/Documents/qmcpack/src/QMCWaveFunctions/ElectronGas/ElectronGasOrbitalBuilder.cpp:23:
/home/svh/Documents/qmcpack/src/Platforms/CUDA/cuBLAS.hpp:22:10: fatal error: cublas_v2.h: No such file or directory
   22 | #include <cublas_v2.h>
      |          ^~~~~~~~~~~~~
compilation terminated.

[ye-luo]

diff --git a/src/Platforms/CUDA/CMakeLists.txt b/src/Platforms/CUDA/CMakeLists.txt
index f84c63ee1..ca1dd0092 100644
--- a/src/Platforms/CUDA/CMakeLists.txt
+++ b/src/Platforms/CUDA/CMakeLists.txt
@@ -16,7 +16,7 @@ if(NOT QMC_CUDA2HIP)
   add_library(platform_cuda_runtime ${CUDA_RT_SRCS})
   target_link_libraries(platform_cuda_runtime PUBLIC CUDA::cudart)
   add_library(platform_cuda_LA ${CUDA_LA_SRCS})
-  target_link_libraries(platform_cuda_LA PRIVATE CUDA::cublas CUDA::cusolver)
+  target_link_libraries(platform_cuda_LA PUBLIC CUDA::cublas CUDA::cusolver)
 else()
   hip_add_library(platform_cuda_runtime ${CUDA_RT_SRCS})
   target_link_libraries(platform_cuda_LA PUBLIC platform_rocm_LA)
@@ -24,4 +24,4 @@ else()
   target_link_libraries(platform_cuda_runtime PUBLIC platform_rocm_runtime)
 endif()
 
-target_link_libraries(platform_cuda_LA PRIVATE platform_cuda_runtime)
+target_link_libraries(platform_cuda_LA PUBLIC platform_cuda_runtime)
yeluo@epyc-server:~/opt/cleanup/qmcpack$ 

Try this.

[ye-luo]

Regarding docs. CUDA_ARCH to CMAKE_CUDA_ARCHITECTURES is the most notable change.

[prckent]

Thanks @ye-luo Re: Summit. I was expecting more weirdness.

The manual might need updating.

The scripts for the nightlies presumably will as well, but we (I) can fix those after this is in.

[quantumsteve]

diff --git a/src/Platforms/CUDA/CMakeLists.txt b/src/Platforms/CUDA/CMakeLists.txt
index f84c63ee1..ca1dd0092 100644
--- a/src/Platforms/CUDA/CMakeLists.txt
+++ b/src/Platforms/CUDA/CMakeLists.txt
@@ -16,7 +16,7 @@ if(NOT QMC_CUDA2HIP)
   add_library(platform_cuda_runtime ${CUDA_RT_SRCS})
   target_link_libraries(platform_cuda_runtime PUBLIC CUDA::cudart)
   add_library(platform_cuda_LA ${CUDA_LA_SRCS})
-  target_link_libraries(platform_cuda_LA PRIVATE CUDA::cublas CUDA::cusolver)
+  target_link_libraries(platform_cuda_LA PUBLIC CUDA::cublas CUDA::cusolver)
 else()
   hip_add_library(platform_cuda_runtime ${CUDA_RT_SRCS})
   target_link_libraries(platform_cuda_LA PUBLIC platform_rocm_LA)
@@ -24,4 +24,4 @@ else()
   target_link_libraries(platform_cuda_runtime PUBLIC platform_rocm_runtime)
 endif()
 
-target_link_libraries(platform_cuda_LA PRIVATE platform_cuda_runtime)
+target_link_libraries(platform_cuda_LA PUBLIC platform_cuda_runtime)
yeluo@epyc-server:~/opt/cleanup/qmcpack$ 

Try this.

No change. I was able to build with the following change to FindCUDAToolkit.cmake

diff --git a/Modules/FindCUDAToolkit.cmake b/Modules/FindCUDAToolkit.cmake
index 92042d23c4..6f0fa47666 100644
--- a/Modules/FindCUDAToolkit.cmake
+++ b/Modules/FindCUDAToolkit.cmake
@@ -853,7 +853,7 @@ if(CUDAToolkit_FOUND)
 
     if (NOT TARGET CUDA::${lib_name} AND CUDA_${lib_name}_LIBRARY)
       add_library(CUDA::${lib_name} UNKNOWN IMPORTED)
-      target_include_directories(CUDA::${lib_name} SYSTEM INTERFACE "${CUDAToolkit_INCLUDE_DIRS}")
+      target_include_directories(CUDA::${lib_name} SYSTEM INTERFACE "${CUDAToolkit_INCLUDE_DIRS}" "${CUDAToolkit_INCLUDE_DIRS}/../../../../../math_libs/include")
       set_property(TARGET CUDA::${lib_name} PROPERTY IMPORTED_LOCATION "${CUDA_${lib_name}_LIBRARY}")
       foreach(dep ${arg_DEPS})
         if(TARGET CUDA::${dep})

[ye-luo]

I was confused by your earlier message. Here is my understanding.

With all the released CMake, CUDA accelerated math libraries shipped inside NVHPC are not well handled by FindCUDAToolkit. In this case, I recommend using CUDAToolkit_ROOT pointing to a standalone CUDA toolkit installation. It is not worth teaching FindCUDAToolkit how to use NVHPC internally shipped CUDA toolkit.
${CUDAToolkit_INCLUDE_DIRS}/../../../../../math_libs/include is only valid for NVHPC internally shipped CUDA toolkit. It should not be needed when CUDAToolkit_ROOT pointing to a standalone CUDA toolkit installation.

Once 3.22 is release, NVHPC shipped CUDA toolkit will be recognized by FindCUDAToolkit.

[quantumsteve]

Once 3.22 is release, NVHPC shipped CUDA toolkit will be recognized by FindCUDAToolkit.

CMake master branch is still broken. I created an issue on CMake gitlab.

I'll reinstall the standalone toolkit locally so these NVHPC issues don't affect me.

edit: updating my PATH and LD_LIBRARY_PATH variables was sufficient.

before:

NVARCH=`uname -s`_`uname -m`; export NVARCH
NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS
MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/21.7/compilers/man; export MANPATH
PATH=$NVCOMPILERS/$NVARCH/21.7/compilers/bin:$PATH; export PATH

export PATH=$NVCOMPILERS/$NVARCH/21.7/comm_libs/mpi/bin:$PATH
export MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/21.7/comm_libs/mpi/man

after:

export PATH=/usr/local/cuda-11.4/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

[markdewing]

This change appears to require more effort to get the right CUDA binaries and include files.
My laptop is Ubuntu 18.04, and the default cuda install is 9.1. I have newer CUDA versions installed in /usr/local/cuda-*.*
I'm using a recent mainline build of llvm for the offload compiler.

Prior to this change, I used the configuration (just the CUDA relevant ones are shown):

-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.3 \
-DCUDA_HOST_COMPILER=`which gcc` \
-DCMAKE_CXX_FLAGS="-I/usr/local/cuda-11.3/include" \
-DENABLE_OFFLOAD=1 \
-DENABLE_CUDA=1 \

To use this PR, the configuration options are

-DCMAKE_CUDA_ARCHITECTURES=61 \
-DCUDAToolkit_ROOT=/usr/local/cuda-11.3 \
-DCMAKE_CXX_FLAGS="-I/usr/local/cuda-11.3/include" \
-DENABLE_OFFLOAD=1 \
-DENABLE_CUDA=1 \

The first problem was it picked up the system nvcc. I changed the path to include /usr/local/cuda-11.3/bin, and it picked up the right nvcc.
But now it appears to be getting the wrong CUDA include file, as in the error below. The right cuda include directory is already given using CMAKE_CXX_FLAGS - I'm not sure how to get it to use the right one.

In file included from /home/mdewing/physics/codes/qmcpack/github/qmcpack/src/QMCWaveFunctions/tests/test_cuBLAS_LU.cpp:18:
/home/mdewing/physics/codes/qmcpack/github/qmcpack/src/Platforms/CUDA/cuBLAS.hpp:253:10: error: no matching function for call to 'cublasSgemmBatched'
  return cublasSgemmBatched(handle, transa, transb, m, n, k, alpha, A, lda, B, ldb, beta, C, ldc, batchCount);
         ^~~~~~~~~~~~~~~~~~
/usr/include/cublas_api.h:2308:39: note: candidate function not viable: 8th argument ('const float *const *') would lose const qualifier
CUBLASAPI cublasStatus_t CUBLASWINAPI cublasSgemmBatched (cublasHandle_t handle,
                                      ^

I'm not certain the previous configuration (develop) works because with the llvm build I have, it crashes in the compiler during compilation.

[PDoakORNL]

I don't think we should have to set the cuda include path by hand, infact that breaks the way the CUDA::xxx targets should work. If a targets dependency on the CUDA targets is expressed properly through its target_link_libraries and possibly its link libaries target_link_libraries the include directory should match the version linked without fail.

[ye-luo]

@markdewing does patch #3492 (comment) work?
CUDAToolkit_ROOT not getting honored is the root of the issue. What is your cmake printout? It should say something about what CUDA being picked up.

[quantumsteve]

@markdewing
What is the value of CUDAToolkit_INCLUDE_DIRS?

Does changing find_package(CUDAToolkit REQUIRED) to find_package(CUDAToolkit 11.3 REQUIRED) make any difference?

[markdewing]

Turns out it was the added include directory that was causing the problem. (-DCMAKE_CXX_FLAGS="-I/usr/local/cuda-11.3/include") It was something I added sometime to fix some problem (NVTX api maybe?), and just left it there. It still seems odd that its presence would have such a large effect.

[ye-luo]

Test this please

[ye-luo]

This PR is ready to be merged except I need to fix a minor scripting issue. If there is no other concerns, I'd like to merge it this afternoon.

[ye-luo]

Test this please

[prckent]

Is it expected that any changes will be needed to the nightly scripts?

[ye-luo]

Is it expected that any changes will be needed to the nightly scripts?

yeluo@ryzen-box:~/opt/qmcpack/tests/test_automation/nightly_test_scripts$ grep CUDA_ *
nightly_anl_bora.sh:    CTEST_FLAGS="$CTEST_FLAGS -DQMC_OPTIONS='-DENABLE_OFFLOAD=ON;-DUSE_OBJECT_TARGET=ON;-DCUDA_HOST_COMPILER=`which gcc`;-DCUDA_PROPAGATE_HOST_FLAGS=OFF;-DQMC_NIO_MAX_SIZE=16'"
nightly_anl_bora.sh:    CTEST_FLAGS="$CTEST_FLAGS -DENABLE_CUDA=1 -DCUDA_ARCH=sm_61"
nightly_anl_bora.sh:    CTEST_FLAGS="$CTEST_FLAGS -DENABLE_CUDA=1 -DCUDA_ARCH=sm_61 -DQMC_OPTIONS='-DQMC_NIO_MAX_SIZE=16'"
nightly_anl_bora.sh:    CTEST_FLAGS="$CTEST_FLAGS -DQMC_CUDA=1 -DCUDA_ARCH=sm_61 -DQMC_OPTIONS='-DQMC_NIO_MAX_SIZE=16'"
nightly_ornl.sh:        CTCFG="$CTCFG -DCMAKE_CXX_FLAGS=-Wno-unknown-cuda-version -DQMC_OPTIONS='-DENABLE_OFFLOAD=ON;-DUSE_OBJECT_TARGET=ON;-DENABLE_CUDA=ON;-DCUDA_ARCH=sm_70;-DCUDA_HOST_COMPILER=`which gcc`'"
nightly_ornl.sh:#export CUDA_VISIBLE_DEVICES=
weekly_ornl.sh:        CTCFG="$CTCFG -DCMAKE_CXX_FLAGS=-Wno-unknown-cuda-version -DQMC_OPTIONS='-DENABLE_OFFLOAD=ON;-DUSE_OBJECT_TARGET=ON;-DENABLE_CUDA=ON;-DCUDA_ARCH=sm_70;-DCUDA_HOST_COMPILER=`which gcc`'"
weekly_ornl.sh:#export CUDA_VISIBLE_DEVICES=

Needs update as
CUDA_ARCH -> CMAKE_CUDA_ARCHITECTURES
CUDA_HOST_COMPILER -> CMAKE_CUDA_HOST_COMPILER

[prckent]

Scripts are now updated. Tests that actually run will be limited due to the recent+unsolved by me cmake/ctest/scripting breakage.