Dot based gemm

[seheracer]

Description:

This PR contains an improved version of KokkosBlas::gemm for CUDA: DotBasedGEMM. DotBasedGEMM implements the optimization for C = betaC + alphaA^TB with A and B matrices both being tall and skinny. C matrix is assumably small, so each entry of C is computed by performing the dot product of respective columns of A and B matrices. Note that the dot products are performed on very long vectors, so each dot product is distributed among multiple teams.

When the conditions of having tall and skinny matrices in the form alpha*A^TB hold, instead of calling CUBLAS' gemm, DotBasedGEMM is called. This is considered as an improvement over CUBLAS' gemm, so DotBasedGEMM never takes place if CUBLAS is not enabled.

Output of test_all_sandia on white (without and with cublas):

../scripts/test_all_sandia --spot-check --arch=Power8,Pascal60

Running on machine: white
Going to test compilers:  gcc/6.4.0 gcc/7.2.0 ibm/16.1.0 cuda/9.2.88 cuda/10.0.130
Testing compiler gcc/6.4.0
Testing compiler gcc/7.2.0
  Starting job gcc-7.2.0-OpenMP-release
  Starting job gcc-6.4.0-OpenMP_Serial-release
  PASSED gcc-7.2.0-OpenMP-release
  Starting job gcc-7.2.0-Serial-release
  PASSED gcc-6.4.0-OpenMP_Serial-release
Testing compiler ibm/16.1.0
  Starting job gcc-7.2.0-OpenMP_Serial-release
  PASSED gcc-7.2.0-Serial-release
Testing compiler cuda/9.2.88
  Starting job ibm-16.1.0-Serial-release
  PASSED gcc-7.2.0-OpenMP_Serial-release
  PASSED ibm-16.1.0-Serial-release
Testing compiler cuda/10.0.130
  Starting job cuda-9.2.88-Cuda_OpenMP-release
  PASSED cuda-9.2.88-Cuda_OpenMP-release
  Starting job cuda-10.0.130-Cuda_Serial-release
  PASSED cuda-10.0.130-Cuda_Serial-release
#######################################################
PASSED TESTS
#######################################################
cuda-10.0.130-Cuda_Serial-release build_time=1124 run_time=283
cuda-9.2.88-Cuda_OpenMP-release build_time=1034 run_time=222
gcc-6.4.0-OpenMP_Serial-release build_time=555 run_time=290
gcc-7.2.0-OpenMP-release build_time=390 run_time=107
gcc-7.2.0-OpenMP_Serial-release build_time=623 run_time=368
gcc-7.2.0-Serial-release build_time=233 run_time=182
ibm-16.1.0-Serial-release build_time=1336 run_time=262
#######################################################
FAILED TESTS
#######################################################

../scripts/test_all_sandia cuda --spot-check --with-cuda-options=enable_lambda --with-tpls=cublas --arch=Power8,Pascal60

Running on machine: white
Going to test compilers:  cuda/9.2.88 cuda/10.0.130
Testing compiler cuda/9.2.88
Testing compiler cuda/10.0.130
  Starting job cuda-9.2.88-Cuda_OpenMP-release
  PASSED cuda-9.2.88-Cuda_OpenMP-release
  Starting job cuda-10.0.130-Cuda_Serial-release
  PASSED cuda-10.0.130-Cuda_Serial-release
#######################################################
PASSED TESTS
#######################################################
cuda-10.0.130-Cuda_Serial-release build_time=1177 run_time=287
cuda-9.2.88-Cuda_OpenMP-release build_time=1206 run_time=223
#######################################################
FAILED TESTS
#######################################################

[ndellingwood]

@seheracer please edit the PR to change the base to the develop branch instead of master, that should clean up a lot of the commit history.

[srajama1]

@seheracer Thanks a lot !

[ndellingwood]

Thanks @seheracer !

[jhux2]

Thank you, @seheracer. Could this also be applied to Trilinos dev so that EMPIRE can use it soon?

[seheracer]

Thank you, @seheracer. Could this also be applied to Trilinos dev so that EMPIRE can use it soon?

@jhux2, I pushed the same PR into Trilinos develop too: trilinos/Trilinos#6226