Compilation error with cuda 11.1: error: a nontype template parameter may not have class type

[eddy16112]

Here is my cmake cmd, I am using gcc-9.4 and cuda-11.1

cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_CXX_COMPILER=/scratch2/wwu/kokkos/bin/nvcc_wrapper -DBUILD_SHARED_LIBS=ON -DCMAKE_CXX_STANDARD=17 ..

Here is the output of cmake

-- The CXX compiler identification is GNU 9.4.0
-- Check for working CXX compiler: /scratch2/wwu/kokkos/bin/nvcc_wrapper
-- Check for working CXX compiler: /scratch2/wwu/kokkos/bin/nvcc_wrapper -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting policy CMP0074 to use <Package>_ROOT variables
-- The project name is: KokkosKernels
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Enabled Kokkos devices: OPENMP;CUDA
CMake Warning at /scratch2/wwu/kokkos/install/lib/cmake/Kokkos/KokkosConfigCommon.cmake:31 (MESSAGE):
  The installed Kokkos configuration does not support CXX extensions.
  Forcing -DCMAKE_CXX_EXTENSIONS=Off
Call Stack (most recent call first):
  /scratch2/wwu/kokkos/install/lib/cmake/Kokkos/KokkosConfig.cmake:57 (INCLUDE)
  CMakeLists.txt:122 (FIND_PACKAGE)


-- Found Kokkos at /scratch2/wwu/kokkos/install/lib/cmake/Kokkos
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda-11.1 (found version "11.1") 

=======================
Kokkos Kernels ETI Types
   Devices:  <Cuda,CudaSpace>;<OpenMP,HostSpace>
   Scalars:  double
   Ordinals: int
   Offsets:  size_t
   Layouts:  LayoutLeft

Kokkos Kernels components
   COMMON:    ON
   BATCHED:   ON
   BLAS:      ON
   GRAPH:     ON
   SPARSE:    ON

Kokkos Kernels TPLs
   CUBLAS:      /usr/local/cuda-11.1/lib64/libcublas.so
   CUSPARSE:    /usr/local/cuda-11.1/lib64/libcusparse.so
=======================

-- Creating ETI files for Blas1_abs
-- Creating ETI files for Blas1_abs_mv
-- Creating ETI files for Blas1_scal
-- Creating ETI files for Blas1_scal_mv
-- Creating ETI files for Blas1_dot
-- Creating ETI files for Blas1_dot_mv
-- Creating ETI files for Blas_gesv
-- Creating ETI files for Blas1_axpby
-- Creating ETI files for Blas1_axpby_mv
-- Creating ETI files for Blas1_update
-- Creating ETI files for Blas1_update_mv
-- Creating ETI files for Blas1_sum
-- Creating ETI files for Blas1_sum_mv
-- Creating ETI files for Blas1_nrm1
-- Creating ETI files for Blas1_nrm1_mv
-- Creating ETI files for Blas1_nrm2w
-- Creating ETI files for Blas1_nrm2w_mv
-- Creating ETI files for Blas1_nrminf
-- Creating ETI files for Blas1_nrminf_mv
-- Creating ETI files for Blas1_iamax
-- Creating ETI files for Blas1_iamax_mv
-- Creating ETI files for Blas1_nrm2
-- Creating ETI files for Blas1_nrm2_mv
-- Creating ETI files for Blas1_mult
-- Creating ETI files for Blas1_mult_mv
-- Creating ETI files for Blas1_reciprocal
-- Creating ETI files for Blas1_reciprocal_mv
-- Creating ETI files for Blas1_rotg
-- Creating ETI files for Blas1_rotmg
-- Creating ETI files for Blas2_gemv
-- Creating ETI files for Blas3_gemm
-- Creating ETI files for Blas3_trsm
-- Creating ETI files for Blas3_trmm
-- Creating ETI files for Blas_trtri
-- Creating ETI files for Graph_color_d1
-- Creating ETI files for Sparse_sptrsv_solve
-- Creating ETI files for Sparse_spmv_struct
-- Creating ETI files for Sparse_spmv_mv_struct
-- Creating ETI files for Sparse_spmv_bsrmatrix
-- Creating ETI files for Sparse_spmv_mv_bsrmatrix
-- Creating ETI files for Sparse_spmv
-- Creating ETI files for Sparse_spmv_mv
-- Creating ETI files for Sparse_spgemm_symbolic
-- Creating ETI files for Sparse_spgemm_numeric
-- Creating ETI files for Sparse_bspgemm_numeric
-- Creating ETI files for Sparse_spgemm_jacobi
-- Creating ETI files for Sparse_spadd_symbolic
-- Creating ETI files for Sparse_spadd_numeric
-- Creating ETI files for Sparse_spiluk_symbolic
-- Creating ETI files for Sparse_spiluk_numeric
-- Creating ETI files for Sparse_sptrsv_symbolic
-- Creating ETI files for Sparse_trsv
-- Creating ETI files for Sparse_gauss_seidel_symbolic
-- Creating ETI files for Sparse_gauss_seidel_numeric
-- Creating ETI files for Sparse_gauss_seidel_apply
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch2/wwu/kokkos-kernels/build

Then when I compile the code, I am getting a lot of same errors:

/scratch2/wwu/kokkos-kernels/blas/tpls/KokkosBlas1_rotg_tpl_spec_decl.hpp(452): error: invalid partial specialization -- class "KokkosBlas::Impl::Rotg<Kokkos::Cuda, Kokkos::View<Kokkos::complex<float>, Kokkos::LayoutRight, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>>, Kokkos::View<float, Kokkos::LayoutRight, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>>, true, true>" is already fully specialized

/scratch2/wwu/kokkos-kernels/blas/tpls/KokkosBlas1_rotg_tpl_spec_decl.hpp(454): error: a nontype template parameter may not have class type

[lucbv]

I will try to reproduced, this is somehow not triggered by our builds...

[eddy16112]

Thanks, here is the full log including kokkos and kokkos-kernels
https://gitlab.com/StanfordLegion/legion/-/jobs/3246259273

[lucbv]

FYI, I have reproduced the issue on our systems and will work on a fix. This is related to the ETI process so should not be too complicated to fix.

[lucbv]

The fix in PR #1578 worked on our systems, let me know if you want to test this before we merge it otherwise it should be picked up by your CI when the PR is merged.

[eddy16112]

I am getting new errors:

/scratch2/wwu/kokkos-kernels/sparse/src/KokkosSparse_spgemm_handle.hpp(238): error: identifier "cusparse_spgemm_handle_t" is undefined

/scratch2/wwu/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_tpl_spec_decl.hpp(126): error: identifier "CUSPARSE_SPMV_ALG_DEFAULT" is undefined

/scratch2/wwu/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_tpl_spec_decl.hpp(140): error: identifier "CUSPARSE_SPMV_CSR_ALG2" is undefined

[lucbv]

Okay, I am merging the current PR #1578 and will have a look at the error reported above, these are new and I am not sure who modified that code recently?

[vqd8a]

@lucbv @eddy16112 The error identifier "cusparse_spgemm_handle_t" is undefined could be similar to the error in #1564 .

[lucbv]

Thanks @vqd8a for pointing this issue out, I will check with Brian on his PR status, assuming that it only fixes the spgemm side of the problem we still need to look a bit at the spmv issue.