Nightly Trilinos build failure: spmv-related changes incompatible with Stokhos

[ndellingwood]

A recent PR with changes that would impact routines from the ArithTraits class introduced incompatibilities with the Stokhos package in Trilinos:

Sample output:

<https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_Blake_Intel181163_Serial_Cplx/ws/Trilinos/kokkos-kernels/src/sparse/KokkosSparse_spmv.hpp>:1060: undefined reference to `Kokkos::Details::ArithTraits<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Serial> > >::zero()'
<https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_Blake_Intel181163_Serial_Cplx/ws/Trilinos/kokkos-kernels/src/sparse/KokkosSparse_spmv.hpp>:1065: undefined reference to `Kokkos::Details::ArithTraits<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Serial> > >::zero()'
<https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_Blake_Intel181163_Serial_Cplx/ws/Trilinos/kokkos-kernels/src/sparse/KokkosSparse_spmv.hpp>:1066: undefined reference to `Kokkos::Details::ArithTraits<Sacado::MP::Vector<Stokhos::StaticFixedStorage<int, double, 16, Kokkos::Serial> > >::zero()'

This can be reproduced in a Serial build of Trilinos (kokkos-promotion branch) that enables Stokhos and its tests along with the develop branches of kokkos and kokkos-kernels, the nightly build was tested with intel/18 and openmpi enabled.

[ndellingwood]

I did bisection on kokkos-kernels commits, spmv-related changes from PR #1255 resulted in the undefined references in Stokhos, in addition to the failures in #1291

SHAs tested:
kokkos/kokkos@1ec0d1e
4409775
trilinos/Trilinos@ca382db (kokkos-promotion branch)

Reproducer (blake):

module load intel/compilers/18.1.163 cmake/3.19.3 openmpi/2.1.2/intel/18.1.163

export ARCH_CXX_FLAG="-xCORE-AVX512 -mkl"
export ARCH_C_FLAG="-xCORE-AVX512 -mkl"
export BLAS_LIBRARIES="-mkl;${MKLROOT}/lib/intel64/libmkl_intel_lp64.a;${MKLROOT}/lib/intel64/libmkl_intel_thread.a;${MKLROOT}/lib/intel64/libmkl_core.a"
export LAPACK_LIBRARIES=${BLAS_LIBRARIES}

# Top Level Configuration Options
TESTS=OFF
EXAMPLES=OFF
SHARED=ON
OPENMP=OFF
PTHREAD=OFF
SERIAL=ON
COMPLEX=ON

cmake -D Trilinos_ENABLE_COMPLEX_DOUBLE=${COMPLEX} -D TPL_ENABLE_Pthread=ON -D Kokkos_ARCH_SKX=ON -D CMAKE_CXX_FLAGS="-g -Wall" -DCMAKE_CXX_STANDARD=14 -D HAVE_CXX_PRAGMA_WEAK:BOOL=OFF -D CMAKE_AR:FILEPATH=/usr/bin/ar -D CMAKE_STRIP:FILEPATH=/usr/bin/strip -D CMAKE_RANLIB:FILEPATH=/usr/bin/ranlib -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON -D Trilinos_ENABLE_DEBUG:BOOL=OFF -D CMAKE_INSTALL_PREFIX:PATH=${TRILINOS_INSTALL_DIR} -D Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON -D CMAKE_BUILD_TYPE:STRING=RELEASE -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF -D CMAKE_SKIP_RULE_DEPENDENCY=ON -D Trilinos_ENABLE_CHECKED_STL:BOOL=OFF -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D BUILD_SHARED_LIBS:BOOL=${SHARED} -D DART_TESTING_TIMEOUT:STRING=500 -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" -D Trilinos_ENABLE_OpenMP=${OPENMP} -D TPL_ENABLE_MPI=ON -D TPL_ENABLE_BinUtils=OFF -D TPL_ENABLE_SuperLU=OFF   -D TPL_SuperLU_LIBRARIES:STRING="${SUPERLU_ROOT}/lib/libsuperlu.a"   -D TPL_SuperLU_INCLUDE_DIRS:STRING="${SUPERLU_ROOT}/include" -D TPL_ENABLE_BLAS=ON  -D TPL_ENABLE_LAPACK=ON  -D TPL_BLAS_LIBRARIES:PATH="${BLAS_LIBRARIES}" -D TPL_LAPACK_LIBRARIES:PATH="${LAPACK_LIBRARIES}" -D TPL_ENABLE_DLlib=ON -D Trilinos_ENABLE_Intrepid2=OFF   -D Intrepid2_ENABLE_EXAMPLES:BOOL=${EXAMPLES}   -D Intrepid2_ENABLE_TESTS:BOOL=${TESTS} -D Trilinos_ENABLE_Kokkos=ON   -D Kokkos_ENABLE_SERIAL=${SERIAL}   -D Kokkos_ENABLE_PTHREAD=${PTHREAD}   -D Kokkos_ENABLE_OPENMP=${OPENMP}  -D Kokkos_ENABLE_TESTS:BOOL=${TESTS} -D Trilinos_ENABLE_KokkosKernels=ON   -D KokkosKernels_ENABLE_TESTS:BOOL=${TESTS} -D Trilinos_ENABLE_Tpetra=ON   -D Tpetra_INST_SERIAL:BOOL=${SERIAL}   -D Tpetra_INST_OPENMP:BOOL=${OPENMP}   -D Tpetra_INST_PTHREAD:BOOL=${PTHREAD} -D Tpetra_ENABLE_TESTS:BOOL=${TESTS}   -D Tpetra_ENABLE_EXAMPLES:BOOL=${EXAMPLES} -DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos   -DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels -DTrilinos_ENABLE_Sacado=ON -DTrilinos_ENABLE_Stokhos=ON -DStokhos_ENABLE_TESTS=ON -DStokhos_ENABLE_EXAMPLES=ON -D CMAKE_CXX_COMPILER=`which mpicxx` -D CMAKE_C_COMPILER=`which mpicc` -D CMAKE_Fortran_COMPILER=`which mpifort` ${TRILINOS_DIR}

[ndellingwood]

Adding @lucbv @brian-kelley @srajama1 @etphipp

[brian-kelley]

@ndellingwood I just pushed a change that fixed this:

--- a/packages/stokhos/src/sacado/kokkos/vector/linalg/Kokkos_CrsMatrix_MP_Vector.hpp
+++ b/packages/stokhos/src/sacado/kokkos/vector/linalg/Kokkos_CrsMatrix_MP_Vector.hpp
@@ -45,6 +45,7 @@
 #include "Sacado_MP_Vector.hpp"
 #include "Kokkos_View_MP_Vector.hpp"
 #include "KokkosSparse_CrsMatrix.hpp"
+#include "Kokkos_ArithTraits_MP_Vector.hpp"
 #include "KokkosSparse_spmv.hpp"
 #include "Kokkos_Blas1_MP_Vector.hpp" // for some utilities

Stokhos is using templated specializations of ArithTraits in SpMV but didn't include the file with those specializations. I'm not sure why #1255 introduced the errors, though.

Edit: Oh, I get it - KokkosSparse_spmv.hpp didn't even use ArithTraits before that PR

[ndellingwood]

@brian-kelley excellent, thanks for the fix and tracking that down!

[ndellingwood]

I'll close after the confirming it is resolved with the next nightly build

[ndellingwood]

The update to the kokkos-promotion branch of Trilinos resolved the build errors with Stokhos in nightly builds; test failures remain documented in another issue #1291, closing this. Thanks @brian-kelley !