Add HIPManagedSpace support #2079

brian-kelley · 2023-12-21T01:25:40Z

This was requested because Tpetra is adding option to portably use shared space, see trilinos/Trilinos#12622 . So we should be able to instantiate for SharedSpace on HIP (like we do for CUDA and SYCL).

CMake option for ETI
Run unit tests with a Kokkos::Device, not just Kokkos::HIP
- Like we do for Cuda
- Still use HIPSpace unless Managed is the only enabled memspace
Couple of minor fixes
- Allow querying free HIPManagedSpace memory for SpGEMM
- Disable VBD coloring (not a huge deal, had to do same on CUDA)
- Use correct memory space in supernodal SpTRSV solve

Tested by building and testing these two configurations on Caraway:

../kokkos-kernels/cm_generate_makefile.bash \
    --cxxstandard=17 \
    --compiler=hipcc \
    --with-devices=hip \
    --release --kokkos-release \
    --compiler=hipcc \
    --arch=VEGA908 \
    --with-tpls=rocblas,rocsparse

../kokkos-kernels/cm_generate_makefile.bash \
    --cxxstandard=17 \
    --compiler=hipcc \
    --with-devices=hip \
    --release --kokkos-release \
    --compiler=hipcc \
    --arch=VEGA908 \
    --with-tpls=rocblas,rocsparse \
    --cmake-flags="-DKokkosKernels_INST_MEMSPACE_HIPSPACE=OFF -DKokkosKernels_INST_MEMSPACE_HIPMANAGEDSPACE=ON"

- CMake option for ETI - Run unit tests with a Kokkos::Device, not just Kokkos::HIP - Like we do for Cuda - Still use HIPSpace unless Managed is the only enabled memspace - Couple of minor fixes - Allow querying free HIPManagedSpace memory for SpGEMM - Disable VBD coloring (not a huge deal, had to do same on CUDA) - Use correct memory space in SpTRSV solve

kokkos-devops-admin · 2023-12-21T01:37:52Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

kokkos-devops-admin · 2023-12-21T01:43:06Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

Build Num: 1122
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930_Tpls_CLANG13CUDA10

Build Num: 709
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GNU1021

Build Num: 380
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GNU1021_Light_LayoutRight

Build Num: 367
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GNU1021

Build Num: 368
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19_solo

Build Num: 372
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001_solo

Build Num: 344
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

Build Num: 854
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

Build Num: 849
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA90A_ROCM560

Build Num: 843
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA90A_Tpls_ROCM560

Build Num: 364
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (brian-kelley/kokkos-kernels)

Pull Request Author: brian-kelley

kokkos-devops-admin · 2023-12-21T03:27:24Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

Build Num: 1122
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930_Tpls_CLANG13CUDA10

Build Num: 709
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GNU1021

Build Num: 380
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GNU1021_Light_LayoutRight

Build Num: 367
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GNU1021

Build Num: 368
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19_solo

Build Num: 372
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001_solo

Build Num: 344
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

Build Num: 854
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

Build Num: 849
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA90A_ROCM560

Build Num: 843
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA90A_Tpls_ROCM560

Build Num: 364
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_REPO	https://github.com/brian-kelley/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5868e99`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`3dafbed`
PR_LABELS	enhancement
PULLREQUESTNUM	2079
TEST_REPO_ALIAS	KOKKOSKERNELS

kokkos-devops-admin · 2023-12-21T03:27:41Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

kokkos-devops-admin · 2023-12-21T03:27:48Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-22T03:27:55Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-23T03:37:48Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-24T03:37:52Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-25T03:37:55Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-26T03:47:50Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

kokkos-devops-admin · 2023-12-27T03:27:52Z