Nightly test failure: spgemm unit tests failing on White (Power8)

[ndellingwood]

Following merge of #764 nightly tests began failing on White

Sample failures:

23:22:02 3: [ RUN      ] cuda.sparse_spgemm_double_int_int_TestExecSpace
23:22:02 3: values are different.
23:22:02 3: -6.89e-10 -3.9e-11 6.89e-10 2.21e-09 -6.6e-10 0 9e-12 0 -1.69e-10 -4.55e-10 -9.23e-10 3.9e-11 -1.69e-10 -4.42e-10 1.75e-10 2.89e-10 -3.4e-10 -1.666e-09 -5.78e-10 2.89e-10 ... ... ... -5.152e-09 -3.54e-09 -3.54e-09 7.579e-09 1.026e-09 -3.54e-09 -3.54e-09 7.24e-10 -3.54e-09 -3.54e-09 -1.42737e-06 -1.42737e-06 -1.06507e-07 2.51456e-06 1.17979e-06 -1.42737e-06 -1.40905e-07 1.17979e-06 -4.58122e-07 -1.76274e-07 
23:22:02 3: -6.6e-10 -1.666e-09 9.52e-10 1.75e-10 9e-12 4.83017e-26 3.38e-10 3.9e-11 -6.89e-10 -1.69e-10 -1.69e-10 8.84e-10 6.89e-10 2.21e-09 -3.15333e-26 -4.55e-10 -9.23e-10 -4.42e-10 1.054e-09 -3.9e-11 ... ... ... -1.40905e-07 6.69981e-07 1.12999e-06 -2.83226e-07 -2.0958e-07 9.6e-05 -1.15739e-07 -3.61583e-07 0.001684 -2.53239e-07 2.144e-09 -2.53143e-07 -4.61947e-07 -9.875e-08 -9.875e-08 0.005404 -2.28315e-05 -4.61947e-07 2.144e-09 0.035299 
23:22:02 3: /home/jenkins/white/workspace/KokkosKernels_White_CudaOpenMP_cuda_101_gcc_740/kokkos-kernels/unit_test/sparse/Test_Sparse_spgemm.hpp:453: Failure
23:22:02 3: Value of: correctResult
23:22:02 3:   Actual: false
23:22:02 3: Expected: true
23:22:02 3: KKMEM still has issue 402 bug; C=AA' is incorrect!
23:22:02 3: 
23:22:02 3: [  FAILED  ] cuda.sparse_spgemm_double_int_int_TestExecSpace (1679 ms)
23:22:02 3: [ RUN      ] cuda.sparse_spgemm_double_int_size_t_TestExecSpace
23:22:02 3: values are different.
23:22:02 3: -6.89e-10 -3.9e-11 8.84e-10 0 -6.6e-10 3.38e-10 -4.55e-10 0 9e-12 -1.69e-10 -9.23e-10 -1.69e-10 3.9e-11 -4.42e-10 1.75e-10 2.89e-10 -3.4e-10 9.52e-10 2.89e-10 -5.78e-10 ... ... ... -5.152e-09 -3.54e-09 -3.54e-09 7.579e-09 1.026e-09 -3.54e-09 -3.54e-09 7.24e-10 -3.54e-09 -3.54e-09 -1.42737e-06 2.51456e-06 1.17979e-06 1.17979e-06 -1.76274e-07 -1.76912e-07 -4.69084e-07 -1.42261e-07 -4.61947e-07 -9.875e-08 
23:22:02 3: -6.6e-10 -1.666e-09 9.52e-10 1.75e-10 9e-12 4.83017e-26 2.21e-09 6.89e-10 -6.89e-10 8.84e-10 -1.69e-10 -1.69e-10 3.9e-11 3.38e-10 -4.55e-10 -3.15333e-26 -4.42e-10 1.054e-09 -9.23e-10 -3.9e-11 ... ... ... -1.40905e-07 6.69981e-07 1.12999e-06 -2.83226e-07 -2.0958e-07 9.6e-05 -1.15739e-07 0.001684 -3.61583e-07 -2.53239e-07 2.144e-09 -2.53143e-07 -4.61947e-07 -9.875e-08 -9.875e-08 0.005404 -2.28315e-05 -4.61947e-07 2.144e-09 0.035299 
23:22:02 3: /home/jenkins/white/workspace/KokkosKernels_White_CudaOpenMP_cuda_101_gcc_740/kokkos-kernels/unit_test/sparse/Test_Sparse_spgemm.hpp:453: Failure
23:22:02 3: Value of: correctResult
23:22:02 3:   Actual: false
23:22:02 3: Expected: true
23:22:02 3: KKMEM still has issue 402 bug; C=AA' is incorrect!
23:22:02 3: 
23:22:02 3: [  FAILED  ] cuda.sparse_spgemm_double_int_size_t_TestExecSpace (1666 ms)

Reproducer instructions:
OpenMP gcc/7.2.0

  # Reproducer instructions:
  #   Load modules:
        source /etc/profile.d/modules.sh
        module purge
        module load cmake/3.12.3 gcc/7.2.0

  #     $KOKKOSKERNELS_PATH/cm_generate_makefile.bash --with-devices=OpenMP --arch=Power8 --compiler=/home/projects/ppc64le/gcc/7.2.0/bin/g++ --cxxflags="-O3 -Wall -Wshadow -pedantic -Werror -Wsign-compare -Wtype-limits -Wignored-qualifiers -Wempty-body -Wclobbered -Wuninitialized " --cxxstandard="11" --ldflags=""  --kokkos-path=$KOKKOS_PATH --kokkoskernels-path=$KOKKOSKERNELS_PATH --with-scalars='complex_double' --with-ordinals=int --with-offsets=int,size_t --with-layouts=LayoutLeft --with-tpls=    --with-options= --with-cuda-options= --no-examples 

Cuda+OpenMP cuda/10.1

  # Reproducer instructions:
  #   Load modules:
        source /etc/profile.d/modules.sh
        module purge
        module load cmake/3.12.3 cuda/10.1.105 gcc/7.2.0 ibm/xl/16.1.0

  #     $KOKKOSKERNELS_PATH/cm_generate_makefile.bash --with-devices=Cuda,OpenMP --arch=Power8,Pascal60 --compiler=<https://jenkins-son.sandia.gov/job/KokkosKernels_White_CudaOpenMP_cuda_101_gcc_740/ws/kokkos/bin/nvcc_wrapper> --cxxflags="-O3 -Wall -Wshadow -pedantic -Werror -Wsign-compare -Wtype-limits -Wuninitialized " --cxxstandard="11" --ldflags="" --with-cuda=/home/projects/ppc64le-pwr8-nvidia/cuda/10.1.105 --kokkos-path=$KOKKOS_PATH --kokkoskernels-path=$KOKKOSKERNELS_PATH --with-scalars='double,complex_double' --with-ordinals=int --with-offsets=int,size_t --with-layouts=LayoutLeft --with-tpls=    --with-options= --with-cuda-options= --no-examples

@seheracer can you take a look at these?

Edit: Adding links to nightly tests
https://jenkins-son.sandia.gov/job/KokkosKernels_White_OpenMP_gcc_720-complex_double/163/console
https://jenkins-son.sandia.gov/job/KokkosKernels_White_CudaOpenMP_cuda_101_gcc_740/101/consoleFull
https://jenkins-son.sandia.gov/job/KokkosKernels_White_CudaOpenMP_cuda_92_gcc_720/changes

[seheracer]

@ndellingwood sure, I'm looking at them.