initial implementation of new ETI #526

jjwilke · 2019-12-13T02:11:07Z

No description provided.

jjwilke · 2019-12-13T02:12:05Z

Remove huge number of cpp files
Use CMake system to generate single source file per kernel
Makefile system will no longer work

jjwilke · 2019-12-13T02:15:55Z

New PR instead of canceled #526

srajama1 · 2019-12-13T05:58:10Z

@jjwilke : This is great ! Deleting 6000 files from the repo, what else can we ask for :)

@crtrott : Can you review this, please ?

ndellingwood · 2019-12-13T07:30:48Z

Please make sure to test with Trilinos as well as the cm_test_all_sandia on various testbeds before considering a merge.

jjwilke · 2019-12-13T18:24:16Z

Testing to-dos:

cm_test_all_sandia on kokkos-dev-2
- cuda/10.1 issues, but other CUDA tests fine
- sparse_serial failed, then passed for pthread build: race condition?
- [RESOLVED] GCC Pthread builds are very fast with new ETI. CUDA builds are VERY slow. I wonder if nvcc is not scaling well with the number of template instantiations per file and this would get faster if we generated more ETI files with fewer instantiations.
cm_test_all_sandia on blake
cm_test_all_sandia on mayer
Trilinos promotion test with kokkos/develop on waterman with CUBLAS/CUSPARSE: handful of tests failing
- 65 - TpetraCore_Core_initialize_where_user_initializes_mpi_MPI_4 (Failed)
- 66 - TpetraCore_Core_ScopeGuard_where_user_initializes_mpi_MPI_4 (Failed)
- 69 - TpetraCore_Core_initialize_where_tpetra_initializes_kokkos_MPI_1 (Failed)
- 70 - TpetraCore_Core_ScopeGuard_where_tpetra_initializes_kokkos_MPI_1 (Failed)
- 71 - TpetraCore_Core_initialize_where_user_initializes_kokkos_MPI_1 (Failed)
- 72 - TpetraCore_Core_ScopeGuard_where_user_initializes_kokkos_MPI_1 (Failed)
- 73 - TpetraCore_Core_initialize_where_tpetra_initializes_mpi_and_user_initializes_kokkos_MPI_2 (Failed)
- 74 - TpetraCore_Core_ScopeGuard_where_tpetra_initializes_mpi_and_user_initializes_kokkos_MPI_2 (Failed)
- 93 - TpetraCore_CrsMatrix_UnitTests4_MPI_4 (Failed)
- 180 - TpetraCore_MatrixMatrix_UnitTests_MPI_4 (Failed)
Trilinos promotion test with kokkos/develop on blake (handful of tests failing). The exact same tests are failing on develop - so I don't this this is an ETI thing.
- 28 - TpetraTSQR_Combine_MPI_1 (Failed)
- 29 - TpetraTSQR_SequentialTsqr_contiguousCacheBlocks_MPI_1 (Failed)
- 30 - TpetraTSQR_SequentialTsqr_noncontiguousCacheBlocks_MPI_1 (Failed)
- 31 - TpetraTSQR_KokkosHostTsqr_MPI_1 (Failed)
- 34 - TpetraTSQR_FullTsqr_Accuracy_MPI_4 (Failed)
- 87 - TpetraCore_CrsMatrix_UnitTests4_MPI_4 (Failed)
Trilinos promotion test with kokkos/develop on waterman without CUBLAS/CUSPARSE: handful of tests failing
- 93 - TpetraCore_CrsMatrix_UnitTests4_MPI_4 (Failed)
- 180 - TpetraCore_MatrixMatrix_UnitTests_MPI_4 (Failed)

ndellingwood · 2019-12-13T19:47:08Z

CMakeLists.txt

-SET(EXECSPACE_SERIAL_VALID_MEM_SPACES    HBWSPACE HOSTSPACE CUDAUVMSPACE)
-SET(EXECSPACE_OPENMP_VALID_MEM_SPACES    HBWSPACE HOSTSPACE CUDAUVMSPACE)
-SET(EXECSPACE_PTHREAD_VALID_MEM_SPACES   HBWSPACE HOSTSPACE CUDAUVMSPACE)
+SET(EXECSPACE_SERIAL_VALID_MEM_SPACES    HBWSPACE HOSTSPACE)


@jjwilke do these changes for valid mem spaces need to propagate to develop independent of this PR?

No, this is only an ETI thing.

so we're double-instantiating all those kernels? sadness :(

I think we were before, too? This PR is a "preserve all existing behavior". The next PR would do the anon spaces stuff.

One thing at a time!

@jjwilke praise Turing and pass the ammunition ;-)

jjwilke · 2019-12-13T20:36:28Z

I had to make quite a few changes for blake to various things, independent of the ETI.

ndellingwood · 2019-12-13T20:39:06Z

I had to make quite a few changes for blake to various things, independent of the ETI.

@jjwilke can you cherry-pick these changes for a separate PR?

jjwilke · 2019-12-14T00:28:17Z

Cherry-picked changes here: #533

jjwilke · 2019-12-15T03:01:50Z

Timing data on kokkos-dev-2, make -j40

Old ETI w/ TESTS
-real 2m38.381s
-user 42m4.556s
-sys 8m11.060s
Old ETI w/o TESTS
-real 1m56.462s
-user 28m27.473s
-sys 5m34.347s
New ETI w/ TESTS
-real 3m34.466s
-user 32m35.050s
-sys 3m58.608s
New ETI w/o TESTS
-real 2m49.847s
-user 18m18.393s
-sys 1m18.796s
New ETI w/ TESTS, one file per ETI instance
-real 1m56.433s
-user 44m59.249s
-sys 6m22.456s
New ETI w/o TESTS one file per ETI instance
-real 1m20.096s
-user 31m46.109s
-sys 3m44.812s

ndellingwood · 2020-01-07T02:05:19Z

Unsuccessful test on kokkos-dev-2, spot-check with cm_test_all_sandia script

#######################################################
PASSED TESTS
#######################################################
#######################################################
FAILED TESTS
#######################################################
clang-8.0-Cuda_OpenMP-release (build failed)
clang-8.0-Pthread_Serial-release (build failed)
cuda-10.1-Cuda_OpenMP-release (build failed)
gcc-7.3.0-OpenMP-release (build failed)
gcc-7.3.0-Pthread-release (build failed)
gcc-9.1-OpenMP-release (build failed)
gcc-9.1-Serial-release (build failed)
intel-18.0.5-OpenMP-release (build failed)

Looks like linking errors with the trsm update:

Testing compiler gcc/7.3.0
  Starting job gcc-7.3.0-OpenMP-release
...
[ 97%] Linking CXX executable KokkosKernels_blas_openmp
CMakeFiles/KokkosKernels_blas_openmp.dir/openmp/Test_OpenMP_Blas3_trsm.cpp.o: In function `void KokkosBlas::trsm<Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::OpenMP>, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::OpenMP> >(char const*, char const*, char const*, char const*, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::OpenMP>::const_value_type&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::OpenMP> const&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::OpenMP> const&)':
Test_OpenMP_Blas3_trsm.cpp:(.text._ZN10KokkosBlas4trsmIN6Kokkos4ViewIPPdJNS1_10LayoutLeftENS1_6OpenMPEEEES7_EEvPKcS9_S9_S9_RNT0_16const_value_typeERKT_RKSA_[_ZN10KokkosBlas4trsmIN6Kokkos4ViewIPPdJNS1_10LayoutLeftENS1_6OpenMPEEEES7_EEvPKcS9_S9_S9_RNT0_16const_value_typeERKT_RKSA_]+0x55a): undefined reference to `KokkosBlas::Impl::TRSM<Kokkos::View<double const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, false, true>::trsm(char const*, char const*, char const*, char const*, double const&, Kokkos::View<double const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&)'
collect2: error: ld returned 1 exit status

....
  Starting job gcc-7.3.0-Pthread-release
...
[ 97%] Linking CXX executable KokkosKernels_blas_threads
CMakeFiles/KokkosKernels_blas_threads.dir/threads/Test_Threads_Blas3_trsm.cpp.o: In function `void KokkosBlas::trsm<Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Threads>, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Threads> >(char const*, char const*, char const*, char const*, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Threads>::const_value_type&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Threads> const&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Threads> const&)':
Test_Threads_Blas3_trsm.cpp:(.text._ZN10KokkosBlas4trsmIN6Kokkos4ViewIPPdJNS1_10LayoutLeftENS1_7ThreadsEEEES7_EEvPKcS9_S9_S9_RNT0_16const_value_typeERKT_RKSA_[_ZN10KokkosBlas4trsmIN6Kokkos4ViewIPPdJNS1_10LayoutLeftENS1_7ThreadsEEEES7_EEvPKcS9_S9_S9_RNT0_16const_value_typeERKT_RKSA_]+0x55a): undefined reference to `KokkosBlas::Impl::TRSM<Kokkos::View<double const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Threads, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Threads, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, false, true>::trsm(char const*, char const*, char const*, char const*, double const&, Kokkos::View<double const**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Threads, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&, Kokkos::View<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Threads, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&)'

etc

jjwilke · 2020-01-07T17:25:03Z

Yeah, somehow I managed to rebase with no conflicts without actually adding trsm to the new ETI system. Anyhow, ETI files are now generated for trsm, which should clear the error. The revision history got refactored to read a bit more sensibly - so you'll need a clean checkout.

On both develop and the branch, I'm getting gemv errors in the tests, though.

ndellingwood · 2020-01-07T22:22:25Z

Tests on kokkos-dev-2 look better:

#######################################################
PASSED TESTS
#######################################################
clang-8.0-Pthread_Serial-release build_time=48 run_time=133
cuda-10.1-Cuda_OpenMP-release build_time=151 run_time=123
gcc-7.3.0-OpenMP-release build_time=46 run_time=46
gcc-7.3.0-Pthread-release build_time=42 run_time=67
gcc-9.1-OpenMP-release build_time=51 run_time=46
gcc-9.1-Serial-release build_time=45 run_time=62
intel-18.0.5-OpenMP-release build_time=129 run_time=45
#######################################################
FAILED TESTS
#######################################################
clang-8.0-Cuda_OpenMP-release (build failed)

The failing clang build is due to -Werror triggered in the fallback gemm routine; talked to Christian about this, we're going to disable checking for the particular warning in cm_test_all_sandia, PR #558 has the change to script, which allows this test to pass.

ndellingwood · 2020-01-08T07:10:38Z

White spot-check testing with cm_test_all_sandia looks good.

gcc tests passed, I had to restart tests after with corrected arch options, below is output for xl and cuda:

${KOKKOSKERNELS_PATH}/scripts/cm_test_all_sandia ibm cuda --kokkos-path=${KOKKOS_PATH} --kokkoskernels-path=${KOKKOSKERNELS_PATH} --spot-check --num=1 --arch=Power8,Pascal60

Running on machine: white
Repository Status:  94ae52daca2f919cb4c9f61afbecaca7eca93a0b completely remove all ETI scripts and raw Makefiles


Going to test compilers:  ibm/16.1.0 cuda/9.2.88
Testing compiler ibm/16.1.0
  Starting job ibm-16.1.0-OpenMP-release
kokkos options:
kokkos devices: OpenMP
kokkos cxx: -O3 -Wall -Wshadow -pedantic -Wsign-compare -Wtype-limits -Wuninitialized
  PASSED ibm-16.1.0-OpenMP-release
  Starting job ibm-16.1.0-Serial-release
kokkos options:
kokkos devices: Serial
kokkos cxx: -O3 -Wall -Wshadow -pedantic -Wsign-compare -Wtype-limits -Wuninitialized
  PASSED ibm-16.1.0-Serial-release
Testing compiler cuda/9.2.88
  Starting job ibm-16.1.0-OpenMP_Serial-release
kokkos options:
kokkos devices: OpenMP,Serial
kokkos cxx: -O3 -Wall -Wshadow -pedantic -Wsign-compare -Wtype-limits -Wuninitialized
  PASSED ibm-16.1.0-OpenMP_Serial-release
  Starting job cuda-9.2.88-Cuda_OpenMP-release
kokkos options:
kokkos devices: Cuda,OpenMP
kokkos cxx: -O3 -Wall -Wshadow -pedantic -Werror -Wsign-compare -Wtype-limits -Wuninitialized
  PASSED cuda-9.2.88-Cuda_OpenMP-release

==============================================
Complete.
==============================================
  Starting job cuda-9.2.88-Cuda_Serial-release
kokkos options:
kokkos devices: Cuda,Serial
kokkos cxx: -O3 -Wall -Wshadow -pedantic -Werror -Wsign-compare -Wtype-limits -Wuninitialized
  PASSED cuda-9.2.88-Cuda_Serial-release

srajama1 · 2020-01-08T16:53:23Z

Timing data on kokkos-dev-2, make -j40

* Old ETI w/ TESTS
  -real	2m38.381s
  -user	42m4.556s
  -sys	8m11.060s

* Old ETI w/o TESTS
  -real	1m56.462s
  -user	28m27.473s
  -sys	5m34.347s

* New ETI w/ TESTS
  -real	3m34.466s
  -user	32m35.050s
  -sys	3m58.608s

* New ETI w/o TESTS
  -real	2m49.847s
  -user	18m18.393s
  -sys	1m18.796s

* New ETI w/ TESTS, one file per ETI instance
  -real	1m56.433s
  -user	44m59.249s
  -sys	6m22.456s

* New ETI w/o TESTS one file per ETI instance
  -real	1m20.096s
  -user	31m46.109s
  -sys	3m44.812s

We pushed the one file per ETI instance change, right ? Thanks for keeping track of the times.

jjwilke · 2020-01-08T17:06:42Z

Yes, the most recent commit does one ETI instance per file for better parallelism.

srajama1 · 2020-01-08T17:07:21Z

@jjwilke Thank you !

srajama1 · 2020-01-31T18:44:14Z

@jjwilke Is this ready to go in to develop ?

ndellingwood · 2020-01-31T18:46:36Z

@srajama1 we need to convert testing to the cm_* scripts first, or else nearly all our nightlies will begin failing.

jjwilke · 2020-01-31T19:39:29Z

I believe this is ready. Obviously we'll want some extensive testing.

ndellingwood · 2020-02-06T15:38:39Z

@jjwilke the nightlies are now testing with the cmake build system, so that blocker is removed from this PR.
@srajama1 @iyamazaki when this PR is ready for merge the Makefile build system will be removed, so a cmake configure to build the supernodal sptrsv will be needed to exercise the capability.

jjwilke · 2020-02-10T17:21:10Z

I would merge the TPL PR #546, then merge this. I this branch has been rebased and should be ready to merge. Someone will want to make sure all the recent SPILUK, SPTRSV changes are still working in this branch.

jjwilke · 2020-03-12T16:53:42Z

@ndellingwood spot-check and spot-check-tpls are all passing on kokkos-dev-2. I think this is ready to go.

ndellingwood

lgtm based on passing tests and seeming to be safe with last round of integration testing, this is awesome @jjwilke ! There were a lot of files to scroll down, did I miss instructions for how devs should update the build system when adding new capabilities? That can be added later if not in this PR

ndellingwood · 2020-03-12T18:25:27Z

src/impl/generated_specializations_cpp/change

@@ -0,0 +1,36 @@
+#! /usr/bin/env python


I don't write python code often, wanted to check if this is python 2 vs 3 agnostic?

ndellingwood · 2020-03-12T18:35:43Z

@jjwilke arg, I dropped another merge conflict on you, sorry...

ndellingwood · 2020-03-13T15:21:07Z

@jjwilke if you have time to clean up the merge conflict (sorry for another) I think we're good to merge this in?

jjwilke · 2020-03-13T15:41:41Z

Rebased. Ready to go?

ndellingwood · 2020-03-17T20:08:05Z

Merging that was like our version of "the snap"

jjwilke mentioned this pull request Dec 13, 2019

Conversion to new ETI requiring minimal C++ code additions #496

Closed

srajama1 requested a review from crtrott December 13, 2019 05:57

ndellingwood reviewed Dec 13, 2019

View reviewed changes

jjwilke force-pushed the new-eti-system branch from 7791c91 to 858300f Compare December 15, 2019 18:29

jjwilke mentioned this pull request Dec 17, 2019

BLAS and cuBLAS TPL interface for TRSM #532

Merged

jjwilke force-pushed the new-eti-system branch from 858300f to 8b2b5c8 Compare January 6, 2020 23:04

jjwilke force-pushed the new-eti-system branch from 8b2b5c8 to 94ae52d Compare January 7, 2020 17:21

jjwilke force-pushed the new-eti-system branch from 94ae52d to bf2038e Compare February 7, 2020 18:24

jjwilke mentioned this pull request Feb 10, 2020

Initial support for SuperLU TPL: Issue #545 #546

Closed

jjwilke force-pushed the new-eti-system branch 2 times, most recently from 7906fb0 to b0e1e79 Compare March 12, 2020 16:53

ndellingwood approved these changes Mar 12, 2020

View reviewed changes

jjwilke and others added 3 commits March 13, 2020 08:36

initial implementation of new ETI

a05bfd5

speedup build with single instance per cpp file

cf2d55c

completely remove all ETI scripts and raw Makefiles

5017f1e

jjwilke force-pushed the new-eti-system branch from b0e1e79 to 5017f1e Compare March 13, 2020 15:40

ndellingwood merged commit e5b09a0 into kokkos:develop Mar 17, 2020

kokkos-devops-admin mentioned this pull request May 20, 2024

Interface for LAPACK geqrf() #2205

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial implementation of new ETI #526

initial implementation of new ETI #526

jjwilke commented Dec 13, 2019

jjwilke commented Dec 13, 2019

jjwilke commented Dec 13, 2019

srajama1 commented Dec 13, 2019

ndellingwood commented Dec 13, 2019

jjwilke commented Dec 13, 2019 •

edited

Loading

ndellingwood Dec 13, 2019

jjwilke Dec 13, 2019

mhoemmen Dec 16, 2019

jjwilke Dec 16, 2019

mhoemmen Dec 16, 2019

jjwilke commented Dec 13, 2019

ndellingwood commented Dec 13, 2019

jjwilke commented Dec 14, 2019

jjwilke commented Dec 15, 2019 •

edited

Loading

ndellingwood commented Jan 7, 2020

jjwilke commented Jan 7, 2020

ndellingwood commented Jan 7, 2020

ndellingwood commented Jan 8, 2020

srajama1 commented Jan 8, 2020

jjwilke commented Jan 8, 2020

srajama1 commented Jan 8, 2020

srajama1 commented Jan 31, 2020

ndellingwood commented Jan 31, 2020

jjwilke commented Jan 31, 2020

ndellingwood commented Feb 6, 2020

jjwilke commented Feb 10, 2020

jjwilke commented Mar 12, 2020

ndellingwood left a comment

ndellingwood Mar 12, 2020

ndellingwood commented Mar 12, 2020

ndellingwood commented Mar 13, 2020

jjwilke commented Mar 13, 2020

ndellingwood commented Mar 17, 2020

initial implementation of new ETI #526

initial implementation of new ETI #526

Conversation

jjwilke commented Dec 13, 2019

jjwilke commented Dec 13, 2019

jjwilke commented Dec 13, 2019

srajama1 commented Dec 13, 2019

ndellingwood commented Dec 13, 2019

jjwilke commented Dec 13, 2019 • edited Loading

ndellingwood Dec 13, 2019

Choose a reason for hiding this comment

jjwilke Dec 13, 2019

Choose a reason for hiding this comment

mhoemmen Dec 16, 2019

Choose a reason for hiding this comment

jjwilke Dec 16, 2019

Choose a reason for hiding this comment

mhoemmen Dec 16, 2019

Choose a reason for hiding this comment

jjwilke commented Dec 13, 2019

ndellingwood commented Dec 13, 2019

jjwilke commented Dec 14, 2019

jjwilke commented Dec 15, 2019 • edited Loading

ndellingwood commented Jan 7, 2020

jjwilke commented Jan 7, 2020

ndellingwood commented Jan 7, 2020

ndellingwood commented Jan 8, 2020

srajama1 commented Jan 8, 2020

jjwilke commented Jan 8, 2020

srajama1 commented Jan 8, 2020

srajama1 commented Jan 31, 2020

ndellingwood commented Jan 31, 2020

jjwilke commented Jan 31, 2020

ndellingwood commented Feb 6, 2020

jjwilke commented Feb 10, 2020

jjwilke commented Mar 12, 2020

ndellingwood left a comment

Choose a reason for hiding this comment

ndellingwood Mar 12, 2020

Choose a reason for hiding this comment

ndellingwood commented Mar 12, 2020

ndellingwood commented Mar 13, 2020

jjwilke commented Mar 13, 2020

ndellingwood commented Mar 17, 2020

jjwilke commented Dec 13, 2019 •

edited

Loading

jjwilke commented Dec 15, 2019 •

edited

Loading