Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MDF: move most expensive kernels over to hierarchical parallelism #1893

Merged
merged 3 commits into from
Jul 12, 2023
Merged

MDF: move most expensive kernels over to hierarchical parallelism #1893

merged 3 commits into from
Jul 12, 2023

Conversation

tmranse
Copy link
Contributor

@tmranse tmranse commented Jul 7, 2023

This MR includes a couple of performance improvements

  • Previously, many of the kernels had been run on a single thread. Move to hierarchical parallelism to properly utilize GPU
  • Previously, already-factored rows were checked for each row using a reduction on the running permutation list, leading to O(n^2) work per iteration in many of the functors. Move to an unordered set for constant time lookup of already-factored rows

@tmranse
Copy link
Contributor Author

tmranse commented Jul 7, 2023

@lucbv @srajama1 MDF email as discussed in email

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@lucbv lucbv self-requested a review July 7, 2023 20:11
@lucbv lucbv added enhancement performance AT: PRE-TEST INSPECTED Mark this PR as approved for testing. labels Jul 7, 2023
@kokkos-devops-admin kokkos-devops-admin removed the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Jul 7, 2023
@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

  • Build Num: 715
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930_Tpls_CLANG13CUDA10

  • Build Num: 306
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020

  • Build Num: 329
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020_Light_LayoutRight

  • Build Num: 599
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC1020

  • Build Num: 560
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19

  • Build Num: 649
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 702
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

  • Build Num: 503
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

  • Build Num: 498
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_ROCM520

  • Build Num: 498
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_Tpls_ROCM520

  • Build Num: 20
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (tmranse/kokkos-kernels)
  • Branch: tmranse/heirMDF
  • SHA: 0eae1f2
  • Mode: TEST_REPO

Pull Request Author: tmranse

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

  • Build Num: 715
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930_Tpls_CLANG13CUDA10

  • Build Num: 306
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020

  • Build Num: 329
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020_Light_LayoutRight

  • Build Num: 599
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC1020

  • Build Num: 560
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19

  • Build Num: 649
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 702
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

  • Build Num: 503
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

  • Build Num: 498
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_ROCM520

  • Build Num: 498
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_Tpls_ROCM520

  • Build Num: 20
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_REPO https://github.com/tmranse/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA 0eae1f2
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA 19b30d8
PR_LABELS enhancement;performance
PULLREQUESTNUM 1893
TEST_REPO_ALIAS KOKKOSKERNELS

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

4 similar comments
@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ lucbv ]!

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

@lucbv lucbv merged commit 3dce1e3 into kokkos:develop Jul 12, 2023
@tmranse tmranse deleted the tmranse/heirMDF branch July 12, 2023 20:15
@ndellingwood ndellingwood mentioned this pull request Jul 13, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucbv lucbv lucbv approved these changes

Assignees

@tmranse tmranse

Labels
enhancement performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tmranse @kokkos-devops-admin @lucbv