Trilinos nightly test failure, Ifpack2_SGSMT_compare_with_Jacobi_MPI_4 "Dimensions do not match"

[ndellingwood]

The added check for compatible dimensions in spmv #1301 seems to have exposed an incompatibility within Ifpack2 unit test Ifpack2_SGSMT_compare_with_Jacobi_MPI_4:

...
07:57:09   Test multiple threaded SGS sweeps compared w.r.t. Jacobi
07:57:09    Test that code {prec_mt_sgs.initialize ();} does not throw : passed
07:57:09    Test that code {prec_mt_sgs.compute ();} does not throw : passed
07:57:09    Test that code {prec_sgs2.initialize ();} does not throw : passed
07:57:09    Test that code {prec_sgs2.compute ();} does not throw : passed
07:57:09  
07:57:09  p=0: *** Caught standard std::exception of type 'std::runtime_error' :
07:57:09  
07:57:09   KokkosSparse::spmv (Generic): Dimensions do not match: , A: 250 x 250, x: 287 x 1, y: 250 x 1
07:57:09  [FAILED]  (0.0245 sec) SGS_MT_double_int_longlong_JacobiComparison_UnitTest
07:57:09  Location: /home/jenkins/weaver-new/workspace/KokkosEco_Trilinos_Weaver_Gcc720_OpenMP_RelDbg/Trilinos/packages/ifpack2/test/unit_tests/Ifpack2_UnitTestSGSMT.cpp:59
07:57:09  
07:57:09 
07:57:09 The following tests FAILED:
07:57:09     0. SGS_MT_double_int_longlong_JacobiComparison_UnitTest ... 

@brian-kelley can you investigate this failure following #1301?

Reproducer:
This can be reproduced for various build types with Ifpack2 enabled, for example a Serial build on Blake. I'm posting a full configuration script used on Blake, but I think the key options to enable are Ifpack2 and MPI.

kokkos/kokkos@72efefa
4b9e29b
trilinos/Trilinos@05d2d86 (kokkos-promotion branch)

• Trilinos' develop branch can be used as long as Stokhos and downstream packages are disabled

# Modules, environment
export ATDM_CONFIG_REGISTER_CUSTOM_CONFIG_DIR=${TRILINOS_DIR}/cmake/std/atdm/contributed/blake
source ${TRILINOS_DIR}/cmake/std/atdm/load-env.sh intel-serial
export ATDM_CONFIG_USE_NINJA=OFF

export ARCH_CXX_FLAG="-xCORE-AVX512 -mkl"
export ARCH_C_FLAG="-xCORE-AVX512 -mkl"
export BLAS_LIBRARIES="-mkl;${MKLROOT}/lib/intel64/libmkl_intel_lp64.a;${MKLROOT}/lib/intel64/libmkl_intel_thread.a;${MKLROOT}/lib/intel64/libmkl_core.a"
export LAPACK_LIBRARIES=${BLAS_LIBRARIES}

# Top Level Configuration Options
TESTS=ON
EXAMPLES=ON
SHARED=ON

CUDA=OFF
OPENMP=OFF
PTHREAD=OFF
SERIAL=ON
COMPLEX=ON

# Full configuration tested on Blake
cmake \
-DCMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL_DIR}" \
-DCMAKE_CXX_STANDARD="14" \
-D Trilinos_ENABLE_COMPLEX_DOUBLE=${COMPLEX} \
\
-D Kokkos_ARCH_SKX=ON \
-D CMAKE_CXX_FLAGS="-g" \
-D CMAKE_C_FLAGS="${ARCH_C_FLAG} -g" \
-D CMAKE_Fortran_FLAGS="${ARCH_C_FLAG} -g" \
-D CMAKE_EXE_LINKER_FLAGS="${ARCH_C_FLAG}" \
-D CMAKE_Fortran_COMPILER="mpif77" \
-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
-D Trilinos_ENABLE_DEBUG:BOOL=OFF \
\
-D Trilinos_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON \
-D CMAKE_BUILD_TYPE:STRING=RELEASE \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
-D CMAKE_SKIP_RULE_DEPENDENCY=ON \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
-D BUILD_SHARED_LIBS:BOOL=${SHARED} \
-D DART_TESTING_TIMEOUT:STRING=500 \
\
-D Trilinos_ENABLE_OpenMP=${OPENMP} \
-D TPL_ENABLE_CUDA=${CUDA} \
-D TPL_ENABLE_MPI=ON \
  -D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-bind-to;socket;-map-by;socket" \
-D TPL_ENABLE_BinUtils=OFF \
-D TPL_ENABLE_SuperLU=OFF \
  -D TPL_SuperLU_LIBRARIES:STRING="${SUPERLU_ROOT}/lib/libsuperlu.a" \
  -D TPL_SuperLU_INCLUDE_DIRS:STRING="${SUPERLU_ROOT}/include" \
-D TPL_ENABLE_BLAS=ON \
  -D TPL_BLAS_LIBRARIES:PATH="${BLAS_LIBRARIES}" \
-D TPL_ENABLE_LAPACK=ON \
  -D TPL_LAPACK_LIBRARIES:PATH="${LAPACK_LIBRARIES}" \
-D TPL_ENABLE_Boost=ON \
   -D Boost_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D Boost_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_BoostLib=ON \
   -D BoostLib_INCLUDE_DIRS:PATH="${BOOST_ROOT}/include" \
   -D BoostLib_LIBRARY_DIRS:PATH="${BOOST_ROOT}/lib" \
-D TPL_ENABLE_Netcdf=ON \
  -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF_ROOT}/include" \
  -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF_ROOT}/lib" \
  -D TPL_Netcdf_LIBRARIES:PATH="${NETCDF_ROOT}/lib/libnetcdf.a;${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a;${PNETCDF_ROOT}/lib/libpnetcdf.a" \
  -D TPL_Netcdf_PARALLEL:BOOL=ON \
-D TPL_ENABLE_HDF5=ON \
  -D HDF5_INCLUDE_DIRS:PATH="${HDF5_ROOT}/include" \
  -D TPL_HDF5_LIBRARIES:PATH="${HDF5_ROOT}/lib/libhdf5_hl.a;${HDF5_ROOT}/lib/libhdf5.a;${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_Zlib=ON \
  -D Zlib_INCLUDE_DIRS:PATH="${ZLIB_ROOT}/include" \
  -D TPL_Zlib_LIBRARIES:PATH="${ZLIB_ROOT}/lib/libz.a" \
-D TPL_ENABLE_DLlib=ON \
\
-D Trilinos_ENABLE_Amesos=ON \
-D Trilinos_ENABLE_Amesos2=ON \
  -D Amesos2_ENABLE_TESTS:BOOL=${TESTS} \
  -D Amesos2_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_ShyLU_NodeTacho=${ENABLE_SHYLU} \
  -D ShyLU_NodeTacho_ENABLE_TESTS:BOOL=${TESTS} \
  -D ShyLU_NodeTacho_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Anasazi=ON \
  -D Anasazi_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_AztecOO=ON \
-D Trilinos_ENABLE_Belos=ON \
  -D Belos_ENABLE_TESTS:BOOL=${TESTS} \
  -D Belos_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Epetra=ON \
-D Trilinos_ENABLE_EpetraExt=ON \
-D Trilinos_ENABLE_Ifpack=ON \
  -D Ifpack_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Ifpack2=ON \
  -D Ifpack2_ENABLE_TESTS:BOOL=${TESTS} \
  -D Ifpack2_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Intrepid=ON \
  -D Intrepid_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
  -D Intrepid_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Intrepid2=ON \
  -D Intrepid2_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
  -D Intrepid2_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Isorropia=ON \
  -D Isorropia_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=${SERIAL} \
  -D Kokkos_ENABLE_PTHREAD=${PTHREAD} \
  -D Kokkos_ENABLE_OPENMP=${OPENMP} \
  -D Kokkos_ENABLE_CUDA=${CUDA} \
  -D Kokkos_ENABLE_CUDA_LAMBDA=${CUDA} \
  -D Kokkos_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_KokkosKernels=ON \
  -D KokkosKernels_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_ML=ON \
  -D ML_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Moertel=OFF \
  -D Moertel_ENABLE_EXAMPLES=OFF \
-D Trilinos_ENABLE_MueLu=ON \
  -D MueLu_INST_DOUBLE_INT_LONGINT:BOOL=ON \
  -D MueLu_ENABLE_TESTS:BOOL=${TESTS} \
  -D MueLu_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
  -D MueLu_ENABLE_Experimental:BOOL=ON \
  -D MueLu_ENABLE_Kokkos_Refactor:BOOL=ON \
  -D MueLu_ENABLE_Epetra:BOOL=OFF \
  -D Trilinos_ENABLE_NOX=ON \
  -D NOX_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Panzer=ON \
  -D Panzer_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Phalanx=ON \
  -D Phalanx_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_ROL=ON \
  -D ROL_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Sacado=ON \
  -D Sacado_ENABLE_UNINIT:BOOL=ON \
  -D Sacado_ENABLE_TESTS:BOOL=${TESTS} \
  -D Sacado_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Stokhos=ON \
  -D Stokhos_ENABLE_TESTS:BOOL=${TESTS} \
  -D Stokhos_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Stratimikos=ON \
  -D Stratimikos_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Thyra=ON \
  -D Thyra_ENABLE_TESTS:BOOL=${TESTS} \
-D Trilinos_ENABLE_Tpetra=ON \
  -D Tpetra_INST_SERIAL:BOOL=${SERIAL} \
  -D Tpetra_INST_OPENMP:BOOL=${OPENMP} \
  -D Tpetra_INST_PTHREAD:BOOL=${PTHREAD} \
  -D Tpetra_INST_CUDA:BOOL=${CUDA} \
  -D Tpetra_ENABLE_TESTS:BOOL=${TESTS} \
  -D Tpetra_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_TrilinosCouplings=ON \
  -D TrilinosCouplings_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
-D Trilinos_ENABLE_Triutils=ON \
-D Trilinos_ENABLE_Xpetra=ON \
  -D Xpetra_ENABLE_TESTS:BOOL=${TESTS} \
  -D Xpetra_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
  -D Xpetra_ENABLE_Experimental:BOOL=ON \
  -D Xpetra_ENABLE_Kokkos_Refactor:BOOL=ON \
-D Trilinos_ENABLE_Zoltan=ON \
-D Trilinos_ENABLE_Zoltan2=ON \
  -D Zoltan2_ENABLE_TESTS:BOOL=${TESTS} \
  -D Zoltan2_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
\
  -D Trilinos_ENABLE_STKMesh:BOOL=ON \
  -D Trilinos_ENABLE_STKSimd:BOOL=ON \
  -D Trilinos_ENABLE_STKIO:BOOL=ON \
  -D Trilinos_ENABLE_STKTransfer:BOOL=ON \
  -D Trilinos_ENABLE_STKSearch:BOOL=ON \
  -D Trilinos_ENABLE_STKUtil:BOOL=ON \
  -D Trilinos_ENABLE_STKTopology:BOOL=ON \
  -D Trilinos_ENABLE_STKClassic:BOOL=OFF \
  -D Trilinos_ENABLE_SEACASExodus:BOOL=ON \
  -D Trilinos_ENABLE_SEACASEpu:BOOL=ON \
  -D Trilinos_ENABLE_SEACASExodiff:BOOL=ON \
  -D Trilinos_ENABLE_SEACASNemspread:BOOL=ON \
  -D Trilinos_ENABLE_SEACASNemslice:BOOL=ON \
  -D Trilinos_ENABLE_SEACASAprepro_lib:BOOL=ON \
$TRILINOS_DIR

For source override, you can add symbolic links to the kokkos and kokkos-kernels repositories in the Trilinos source directory and add the cmake options:
-DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos
-DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \

[ndellingwood]

This update resolved the nightly test failure, thanks @brian-kelley