Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tpl rocblas and rocsparse #1153

Merged
merged 6 commits into from
Nov 1, 2021
Merged

Conversation

lucbv
Copy link
Contributor

@lucbv lucbv commented Oct 26, 2021

Adding support for AMD device libraries: rocBLAS and rocSPARSE.
Everything configures, compiles and runs fine on Spock, there is a chance that this will fail on caraway depending on how the libraries are installed.

@@ -1,514 +1,573 @@
FUNCTION(kokkoskernels_append_config_line LINE)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like autoformatting broke some things in this file - sometimes it inserted a space, likeKOKKOSKERNELS_${TPL_NAME}_ROOT became KOKKOSKERNELS_${TPL_NAME} _ROOT, or changed

${SOME_VAR}

to

${
SOME_VAR}

Copy link
Contributor

@brian-kelley brian-kelley Oct 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Come to think of it, clang-format doesn't actually check the file extension or type. If you pass it a non-C++ file, it will be happy to mangle it :)

Maybe the script to format all the git added files could check for hpp/cpp file extensions ( @e10harvey )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I will look at that again, I want to add something for the rocsparse handle first so that the PR will set us up for integration of some of the TPLs in the blas and sparse kernels.
Thanks for noticing these though, I'll make sure to run clang-format and then build and test on Spock : )

@lucbv lucbv force-pushed the TPL_rocblas_rocsparse branch 2 times, most recently from acf2802 to 7a9e541 Compare October 27, 2021 19:06
@lucbv
Copy link
Contributor Author

lucbv commented Oct 27, 2021

@brian-kelley
I believe this now adds everything I want and it fixes the issues with formatting I had previously.
Assuming the goes well, we can then start adding actual support in our kernels for both libraries : )

@lucbv
Copy link
Contributor Author

lucbv commented Oct 27, 2021

@e10harvey
Looking at the clang-format-check, I am not sure if the suggested change by clang-format actually breaks the code, are we allowed to introduce space for the CMake @variables@?

@e10harvey
Copy link
Contributor

Looking at the clang-format-check, I am not sure if the suggested change by clang-format actually breaks the code, are we allowed to introduce space for the CMake @variables@?

Hi @lucbv, https://cmake.org/cmake/help/latest/command/configure_file.html indicates that the syntax is @VAR@. You can use // clang-format: off and // clang-format: on to avoid this clang-format error.

@lucbv
Copy link
Contributor Author

lucbv commented Oct 27, 2021

@e10harvey thanks, I did not think about the local format de-activation, I will do that!
Thanks

@lucbv lucbv force-pushed the TPL_rocblas_rocsparse branch from fc5d495 to b99dd39 Compare October 27, 2021 19:44
@brian-kelley
Copy link
Contributor

@lucbv Since the TPL logic creates a standard option named like KokkosKernels_ROCBLAS_ROOT (which defaults to ENV{ROCBLAS_ROOT}), could you add this block to cmake/Modules/FindTPLROCBLAS.cmake?

+ELSEIF (KokkosKernels_ROCBLAS_ROOT)
+  KOKKOSKERNELS_FIND_IMPORTED(ROCBLAS INTERFACE
+    LIBRARIES
+      rocblas
+    LIBRARY_PATHS
+      ${KokkosKernels_ROCBLAS_ROOT}/lib
+    HEADERS
+      rocblas.h
+    HEADER_PATHS
+      ${KokkosKernels_ROCBLAS_ROOT}/include
+  )

and analogously this to cmake/Modules/FindTPLROCSPARSE.cmake:

+ELSEIF (KokkosKernels_ROCSPARSE_ROOT)
+  KOKKOSKERNELS_FIND_IMPORTED(ROCSPARSE INTERFACE
+    LIBRARIES
+      rocsparse
+    LIBRARY_PATHS
+      ${KokkosKernels_ROCSPARSE_ROOT}/lib
+    HEADERS
+      rocsparse.h
+    HEADER_PATHS
+      ${KokkosKernels_ROCSPARSE_ROOT}/include
+  )

That made it easy to configure on Caraway by setting ROCBLAS_ROOT/ROCSPARSE_ROOT, as an alternative to using ROCM_PATH (I tested both ways).

@lucbv
Copy link
Contributor Author

lucbv commented Oct 27, 2021

@brian-kelley sure, I'll look into it. Anyway I clearly have something wrong in my logic as the whole thing fails when rocblas/rocsparse are not available x)

lucbv added 6 commits October 28, 2021 18:17
Adding some infrastructure in cmake and tests to enable
rocBLAS and rocSPARSE and check that is compiles correctly
and that simple calls to get library versions are working.
Adding similar utitlies and tests for rocSPARSE
as the ones previously added for rocBLAS.
Since ROCm does not ship with these libraries by default, they should
be turned OFF by default and only turned ON if the user has requested
them explicitely.
Adding KokkosKernels_ROCBLAS_ROOT and KokkosKernels_ROCSPARSE_ROOT
to the FindTPL modules as they are defined automatically by our
TPL system and should be available to users.
@lucbv lucbv force-pushed the TPL_rocblas_rocsparse branch from 8ccbfae to 53537c9 Compare October 29, 2021 00:17
@lucbv
Copy link
Contributor Author

lucbv commented Oct 29, 2021

@brian-kelley
I just pushed the changes for the extra command line options and testing everything on Spock and caraway, I think this should be good to merge.

@lucbv lucbv added the AT: RETEST Have this PR retested. label Oct 29, 2021
@kokkos-devops-admin kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Oct 29, 2021
@lucbv lucbv added the AT: RETEST Have this PR retested. label Oct 29, 2021
@kokkos-devops-admin kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Oct 29, 2021
@@ -0,0 +1,60 @@
// Note: Luc Berger-Vergiat 10/25/21
// Only include this test if compiling
// the cuda sparse tests and cuSPARSE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment should say "HIP/rocSPARSE" probably

Copy link
Contributor

@brian-kelley brian-kelley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. There was just one comment that looked like a copy-paste from the CUDA version of the same file, but no big deal

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job KokkosKernels_PullRequest_Tpls_GCC720_GCC740 to start: Total Wait = 3603

@lucbv lucbv added the AT: RETEST Have this PR retested. label Oct 31, 2021
@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light

  • Build Num: 181
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

  • Build Num: 559
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

  • Build Num: 206
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

  • Build Num: 550
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10

  • Build Num: 181
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

  • Build Num: 539
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_LayoutRight

  • Build Num: 183
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9

  • Build Num: 175
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720_GCC740

  • Build Num: 165
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (lucbv/kokkos-kernels)
  • Branch: TPL_rocblas_rocsparse
  • SHA: 53537c9
  • Mode: TEST_REPO

Pull Request Author: lucbv

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light

  • Build Num: 181
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

  • Build Num: 559
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

  • Build Num: 206
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

  • Build Num: 550
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10

  • Build Num: 181
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

  • Build Num: 539
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_LayoutRight

  • Build Num: 183
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9

  • Build Num: 175
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720_GCC740

  • Build Num: 165
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH TPL_rocblas_rocsparse
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 53537c9
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA eed8d4a
PR_LABELS enhancement;feature request;AT: RETEST
PULLREQUESTNUM 1153
TEST_REPO_ALIAS KOKKOSKERNELS

@kokkos-devops-admin kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Nov 1, 2021
@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ brian-kelley ]!

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

@lucbv lucbv merged commit 2fb2aa5 into kokkos:develop Nov 1, 2021
@lucbv lucbv deleted the TPL_rocblas_rocsparse branch August 31, 2022 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants