BLAS and cuBLAS TPL interface for TRSM #532

vqd8a · 2019-12-13T23:41:38Z

Address the user request in #513 . Only call cuBLAS for CUDA and BLAS for non-CUDA. The KK implementation might be added in another future PR.

mhoemmen

Is it OK to merge this if the fall-back case is not implemented? I think it's OK to have a sequential fall-back, even if that means copying to host and running sequentially (e.g., for CudaSpace Views of element types that cuBLAS does not support).

src/blas/KokkosBlas3_trsm.hpp

src/blas/impl/KokkosBlas3_trsm_impl.hpp

src/blas/impl/KokkosBlas3_trsm_spec.hpp

vqd8a · 2019-12-16T18:58:55Z

@mhoemmen Thanks for your comments. I think the user is OK to have only TPL interfaces in the 3.0 release (#513). Add @iyamazaki for this. I can cover the sequential fall-back in another PR.

src/blas/impl/KokkosBlas3_trsm_spec.hpp

iyamazaki · 2019-12-16T21:47:33Z

Thank you @vqd8a, and @mhoemmen, I want to use this for CholQR in our native Tpetra solvers (which currently copies the basis vectors to the host, and calls BLAS::TRSM).

mhoemmen · 2019-12-17T06:14:28Z

@iyamazaki wrote:

I want to use this for CholQR in our native Tpetra solvers (which currently copies the basis vectors to the host, and calls BLAS::TRSM).

Sure, that's fine. I already have cuSOLVER wrapped for GEQRF in my branch; it's almost ready. I actually have a CholQR implementation in there. I haven't tried it in a while, but it should be straightforward to revive.

srajama1 · 2019-12-17T06:27:35Z

It would be great to have have GEQRF interface in Kokkos Kernels as well. Looking forward to it !

mhoemmen · 2019-12-17T18:29:21Z

I would be happy if kokkos-kernels imitated the design. In particular, I was careful not to expose any TPL headers or types in TSQR header files. That's important especially because of the cuBLAS v1 vs. v2 header compatibility problem that we've seen before. It's also important not to put extern "C" declarations for BLAS functions in header files, since they might conflict with applications' own extern "C" declarations.

vqd8a · 2019-12-17T18:33:38Z

@mhoemmen I am working on adding the sequential fall-back implementation now. Almost done. Needs to change unit tests though since I used BLAS TRMM to generate data.

vqd8a · 2019-12-17T18:34:34Z

It would be great to have have GEQRF interface in Kokkos Kernels as well. Looking forward to it !

@srajama1 I can work on this GEQRF interface after this PR.

jjwilke · 2019-12-17T20:42:53Z

src/impl/tpls/KokkosBlas3_trsm_tpl_spec_avail.hpp

+ KOKKOSBLAS3_TRSM_TPL_SPEC_AVAIL_BLAS( Kokkos::complex<double>, Kokkos::LayoutLeft, Kokkos::LayoutLeft, Kokkos::HostSpace)
+#endif
+#if defined (KOKKOSKERNELS_INST_KOKKOS_COMPLEX_FLOAT_) \
+ && defined (KOKKOSKERNELS_INST_LAYOUTLEFT)


We need to figure out which order this PR and PR #526 should go in. This would affect how this file gets implemented.

@jjwilke This file is only for TPL. I do not think it will be affected by the new ETI implementation.

unit_test/Makefile

srajama1 · 2019-12-18T06:46:17Z

@vqd8a Please talk to @mhoemmen to address the requirements mentioned above when you start on GEQRF.

ndellingwood · 2019-12-19T07:32:00Z

@vqd8a is this PR ready for merge?

vqd8a · 2019-12-19T17:50:43Z

@ndellingwood I am still checking on the implementation of conjugate transpose.

…ngly

vqd8a · 2019-12-19T23:42:48Z

@mhoemmen @ndellingwood I added the sequential fall-back implementation. If it is okay, let me know and I can run spotcheck.

ndellingwood · 2019-12-19T23:45:24Z

@vqd8a doesn't hurt to start a spot-check, please also run tests with tpls enabled and disabled to confirm behavior is as expected. Thanks!

vqd8a · 2019-12-19T23:45:55Z

@ndellingwood one question: this PR now have fall-back implementation, should I need to generate ETI files and add them too or should I wait for @jjwilke new ETI implementation?

vqd8a · 2019-12-19T23:53:17Z

@ndellingwood If I run tests, I have to generate these ETI files using the old script. and do I need to push those files to this PR?

src/blas/impl/KokkosBlas3_trsm_impl.hpp

…s_trsm

vqd8a · 2019-12-20T17:20:40Z

Running spotchecks now.

Author addressed my concern about __restrict__.

vqd8a · 2019-12-20T20:07:38Z

One spotcheck (with BLAS TPL) passed. Two more are running

../kokkos-kernels/scripts/test_all_sandia gcc --spot-check --with-tpls=blas

Running on machine: white
Going to test compilers:  gcc/6.4.0 gcc/7.2.0 gcc/7.4.0
Testing compiler gcc/6.4.0
Testing compiler gcc/7.2.0
  Starting job gcc-7.2.0-OpenMP-release
  Starting job gcc-6.4.0-OpenMP_Serial-release
  PASSED gcc-7.2.0-OpenMP-release
  Starting job gcc-7.2.0-Serial-release
  PASSED gcc-6.4.0-OpenMP_Serial-release
Testing compiler gcc/7.4.0
  Starting job gcc-7.2.0-OpenMP_Serial-release
  PASSED gcc-7.2.0-Serial-release
  Starting job gcc-7.4.0-OpenMP-release
  PASSED gcc-7.4.0-OpenMP-release
  PASSED gcc-7.2.0-OpenMP_Serial-release
#######################################################
PASSED TESTS
#######################################################
gcc-6.4.0-OpenMP_Serial-release build_time=580 run_time=829
gcc-7.2.0-OpenMP-release build_time=414 run_time=317
gcc-7.2.0-OpenMP_Serial-release build_time=591 run_time=723
gcc-7.2.0-Serial-release build_time=241 run_time=449
gcc-7.4.0-OpenMP-release build_time=417 run_time=307
#######################################################
FAILED TESTS
#######################################################

ndellingwood · 2019-12-20T20:21:59Z

@vqd8a just to check, does the spot-check include the fallback or just the tpls, and if the fallback is tested does this work with/without ETI files?

vqd8a · 2019-12-20T20:35:21Z

@ndellingwood This one was just with BLAS TPL. I have two others running (one with CUBLAS and one with the fallback). For the fallback, I have to generate the ETI files. So, for merging, should I push the ETI files here too?

ndellingwood · 2019-12-20T20:58:40Z

@vqd8a I just checked with @jjwilke, go ahead and merge in the ETI files here.

vqd8a · 2019-12-20T21:13:06Z

Thanks @ndellingwood @jjwilke

Spotcheck with cuBLAS passed:

../kokkos-kernels/scripts/test_all_sandia cuda --spot-check --arch=Power8,Pascal60 --with-cuda-options=enable_lamda --with-tpls=cublas

Running on machine: white
Going to test compilers:  cuda/9.2.88 cuda/10.0.130
Testing compiler cuda/9.2.88
Testing compiler cuda/10.0.130
  Starting job cuda-9.2.88-Cuda_OpenMP-release
  PASSED cuda-9.2.88-Cuda_OpenMP-release
  Starting job cuda-10.0.130-Cuda_Serial-release
  PASSED cuda-10.0.130-Cuda_Serial-release
#######################################################
PASSED TESTS
#######################################################
cuda-10.0.130-Cuda_Serial-release build_time=1015 run_time=768
cuda-9.2.88-Cuda_OpenMP-release build_time=1026 run_time=584
#######################################################
FAILED TESTS
#######################################################

vqd8a · 2019-12-20T21:23:59Z

Spotcheck with fall-back implementation passed. ETI files were pushed in.

../kokkos-kernels/scripts/test_all_sandia --spot-check --arch=Power8,Pascal60 --with-cuda-options=enable_lamda

Running on machine: white
Going to test compilers:  gcc/6.4.0 gcc/7.2.0 gcc/7.4.0 ibm/16.1.0 cuda/9.2.88 cuda/10.0.130
Testing compiler gcc/6.4.0
Testing compiler gcc/7.2.0
  Starting job gcc-7.2.0-OpenMP-release
  Starting job gcc-6.4.0-OpenMP_Serial-release
  PASSED gcc-7.2.0-OpenMP-release
  Starting job gcc-7.2.0-Serial-release
  PASSED gcc-7.2.0-Serial-release
Testing compiler gcc/7.4.0
  Starting job gcc-7.2.0-OpenMP_Serial-release
  PASSED gcc-6.4.0-OpenMP_Serial-release
Testing compiler ibm/16.1.0
  Starting job gcc-7.4.0-OpenMP-release
  PASSED gcc-7.4.0-OpenMP-release
Testing compiler cuda/9.2.88
  Starting job ibm-16.1.0-Serial-release
  PASSED gcc-7.2.0-OpenMP_Serial-release
  PASSED ibm-16.1.0-Serial-release
Testing compiler cuda/10.0.130
  Starting job cuda-9.2.88-Cuda_OpenMP-release
  PASSED cuda-9.2.88-Cuda_OpenMP-release
  Starting job cuda-10.0.130-Cuda_Serial-release
  PASSED cuda-10.0.130-Cuda_Serial-release
#######################################################
PASSED TESTS
#######################################################
cuda-10.0.130-Cuda_Serial-release build_time=1052 run_time=846
cuda-9.2.88-Cuda_OpenMP-release build_time=1009 run_time=655
gcc-6.4.0-OpenMP_Serial-release build_time=556 run_time=873
gcc-7.2.0-OpenMP-release build_time=398 run_time=317
gcc-7.2.0-OpenMP_Serial-release build_time=563 run_time=1167
gcc-7.2.0-Serial-release build_time=230 run_time=461
gcc-7.4.0-OpenMP-release build_time=398 run_time=311
ibm-16.1.0-Serial-release build_time=1420 run_time=784
#######################################################
FAILED TESTS
#######################################################

vqd8a · 2019-12-20T21:27:45Z

@ndellingwood Spotchecks passed. If possible, can you merge it?

ndellingwood

Spot-checks have passed, I think @mhoemmen 's comments have been addressed.

vqd8a · 2019-12-20T22:04:01Z

Thanks @ndellingwood for merging.

Add BLAS and cuBLAS interface for TRSM

fe510d2

vqd8a changed the base branch from master to develop December 13, 2019 23:41

vqd8a requested review from kyungjoo-kim, ndellingwood and mhoemmen December 13, 2019 23:42

ndellingwood mentioned this pull request Dec 16, 2019

Need BLAS interface for <>trsm #513

Closed

mhoemmen reviewed Dec 16, 2019

View reviewed changes

src/blas/KokkosBlas3_trsm.hpp Outdated Show resolved Hide resolved

src/blas/KokkosBlas3_trsm.hpp Outdated Show resolved Hide resolved

src/blas/impl/KokkosBlas3_trsm_impl.hpp Outdated Show resolved Hide resolved

src/blas/impl/KokkosBlas3_trsm_spec.hpp Outdated Show resolved Hide resolved

vqd8a added 4 commits December 16, 2019 10:49

Merge branch 'develop' into blas_trsm

1cdc8c2

Fix typo

15302fe

Modify note for KK implementation

c3c89e3

Use double quotes for KK headers and allow RHS with rank 2 only

96ab3d9

mhoemmen reviewed Dec 16, 2019

View reviewed changes

src/blas/impl/KokkosBlas3_trsm_spec.hpp Outdated Show resolved Hide resolved

vqd8a requested a review from srajama1 December 16, 2019 20:03

jjwilke reviewed Dec 17, 2019

View reviewed changes

vqd8a added 3 commits December 19, 2019 16:37

Add sequential fall-back implementation and change unit tests accordi…

fe330f8

…ngly

Accidentally pushed this original file

f2040de

Accidentally pushed this original file

387640c

mhoemmen previously requested changes Dec 20, 2019

View reviewed changes

src/blas/impl/KokkosBlas3_trsm_impl.hpp Outdated Show resolved Hide resolved

vqd8a added 2 commits December 20, 2019 10:15

Use KOKKOS_RESTRICT

85c76eb

Merge branch 'blas_trsm' of github.com:kokkos/kokkos-kernels into bla…

920b213

…s_trsm

vqd8a added 2 commits December 20, 2019 12:00

Fix unused warning

dda5321

Merge branch 'develop' into blas_trsm

686b2bd

Add ETI files

f2394db

ndellingwood approved these changes Dec 20, 2019

View reviewed changes

ndellingwood merged commit d29beba into develop Dec 20, 2019

iyamazaki mentioned this pull request Feb 11, 2020

Use Kokkos-kernels TRSM for CholQR trilinos/Trilinos#6813

Merged

ndellingwood deleted the blas_trsm branch October 29, 2020 16:01

kokkos-devops-admin mentioned this pull request May 22, 2024

Interface for LAPACK geqrf() #2205

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BLAS and cuBLAS TPL interface for TRSM #532

BLAS and cuBLAS TPL interface for TRSM #532

vqd8a commented Dec 13, 2019

mhoemmen left a comment

vqd8a commented Dec 16, 2019 •

edited

Loading

iyamazaki commented Dec 16, 2019

mhoemmen commented Dec 17, 2019

srajama1 commented Dec 17, 2019

mhoemmen commented Dec 17, 2019

vqd8a commented Dec 17, 2019

vqd8a commented Dec 17, 2019

jjwilke Dec 17, 2019

vqd8a Dec 17, 2019

srajama1 commented Dec 18, 2019

ndellingwood commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 19, 2019

ndellingwood commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019 •

edited

Loading

ndellingwood commented Dec 20, 2019

vqd8a commented Dec 20, 2019

ndellingwood commented Dec 20, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019

ndellingwood left a comment

vqd8a commented Dec 20, 2019

BLAS and cuBLAS TPL interface for TRSM #532

BLAS and cuBLAS TPL interface for TRSM #532

Conversation

vqd8a commented Dec 13, 2019

mhoemmen left a comment

Choose a reason for hiding this comment

vqd8a commented Dec 16, 2019 • edited Loading

iyamazaki commented Dec 16, 2019

mhoemmen commented Dec 17, 2019

srajama1 commented Dec 17, 2019

mhoemmen commented Dec 17, 2019

vqd8a commented Dec 17, 2019

vqd8a commented Dec 17, 2019

jjwilke Dec 17, 2019

Choose a reason for hiding this comment

vqd8a Dec 17, 2019

Choose a reason for hiding this comment

srajama1 commented Dec 18, 2019

ndellingwood commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 19, 2019

ndellingwood commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 19, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019 • edited Loading

ndellingwood commented Dec 20, 2019

vqd8a commented Dec 20, 2019

ndellingwood commented Dec 20, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019

vqd8a commented Dec 20, 2019

ndellingwood left a comment

Choose a reason for hiding this comment

vqd8a commented Dec 20, 2019

vqd8a commented Dec 16, 2019 •

edited

Loading

vqd8a commented Dec 20, 2019 •

edited

Loading