Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix SpGEMM crashing on empty rows #1220

Merged
merged 4 commits into from
Dec 8, 2021

Conversation

mzuzek
Copy link

@mzuzek mzuzek commented Dec 8, 2021

Contents

  • SpGEMM Unit test:
    • include SPGEMM_KK_LP to cover MultiCoreTag4 functor in hash method on CPU;
    • include SPGEMM_KK_SPEED to cover speed method - KokkosSPGEMM_numeric_speed();
    • Add case with 0-nnz non-empty arrays (empty rows);
  • SpGEMM fixes:
    • skip spgemm numeric phase altogether if any matrix has zero rows or columns;
    • skip computation for empty rows in GPUTag4 (SPGEMM_KK_MEMORY_SPREADTEAM) and GPUTag6 (SPGEMM_KK_MEMORY_BIGSPREADTEAM)
    • align shared memory size according to Kokkos requirements.

Details

Shared memory alignment

teamMember.team_shmem().get_shmem(shared_memory_size) aligns the size to 8 bytes rounding up and if this number is higher than allocation size in team_shmem_size(), the allocation will fail returning nullptr (and crash follow-up computation). Thus we need to always align our shared_memory_size.

Is this semantic and requirement documented in Kokkos clearly enough ?

Empty row guard

Without proposed guards that skip empty rows, GPUTag4 and GPUTag6 can crash trying to fetch entriesA/valuesA as a_col_begin_offset in case of empty row indexes the first element of the next row and - when A has trailing empty rows - falls outside the view bounds:

      const size_type a_col_begin_offset = row_mapA[row_index];

      nnz_lno_t a_col_ind = entriesA[a_col_begin_offset]; // << index out of bounds
      scalar_t a_col_val  = valuesA[a_col_begin_offset];  // << when A has trailing empty row(s)

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@lucbv lucbv requested review from brian-kelley and lucbv December 8, 2021 19:41
Copy link
Contributor

@lucbv lucbv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good I just want to make sure we are not missing a corner case, also @brian-kelley please look at this too.

@@ -139,6 +139,8 @@ void spgemm_numeric(KernelHandle *handle,
"If you need this case please let kokkos-kernels developers know.\n");
}

if (m < 1 || n < 1 || k < 1) return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not recall the exact interface we provide so I just want to make sure that the short cut above does not create issues for something like: C=beta*C+alpha*A*B which would fail to compute C=beta*C if k==0.
I believe that we only implement C=alpha*A*B though?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, we don't support adding to an existing C, so this check is OK.

@lucbv lucbv added the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Dec 8, 2021
@kokkos-devops-admin kokkos-devops-admin removed the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Dec 8, 2021
@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed.

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

  • Build Num: 16
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

  • Build Num: 16
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

  • Build Num: 674
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

  • Build Num: 321
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

  • Build Num: 665
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

  • Build Num: 652
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 58
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (NexGenAnalytics/kokkos-kernels)
  • Branch: fix-spgemm-empty-row-crash
  • SHA: 7fcea13
  • Mode: TEST_REPO

Pull Request Author: MikolajZuzek

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

  • Build Num: 16
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

  • Build Num: 16
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

  • Build Num: 674
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

  • Build Num: 321
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

  • Build Num: 665
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

  • Build Num: 652
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 58
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH fix-spgemm-empty-row-crash
KOKKOSKERNELS_SOURCE_REPO https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 7fcea13
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 7413f2a
PR_LABELS
PULLREQUESTNUM 1220
TEST_REPO_ALIAS KOKKOSKERNELS

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

Copy link
Contributor

@brian-kelley brian-kelley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @MikolajZuzek !

@@ -139,6 +139,8 @@ void spgemm_numeric(KernelHandle *handle,
"If you need this case please let kokkos-kernels developers know.\n");
}

if (m < 1 || n < 1 || k < 1) return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, we don't support adding to an existing C, so this check is OK.

Copy link
Contributor

@lucbv lucbv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ brian-kelley lucbv ]!

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

@lucbv lucbv merged commit b609e0b into kokkos:develop Dec 8, 2021
@mzuzek mzuzek deleted the fix-spgemm-empty-row-crash branch December 16, 2021 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants