Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF TESTS: adding utilities and instantiation wrapper #1676

Merged
merged 2 commits into from
Feb 10, 2023

Conversation

lucbv
Copy link
Contributor

@lucbv lucbv commented Feb 7, 2023

The goal of this work is to create a common core infrastructure for the performance test in order to simplify maintenance. Here two ideas are introduced:

  1. the instantiation wrapper
  2. the common input parser

both are trying to capture some of the implementation of our performance test in generic functions that can be called instead of duplicating logic around instantiation and command line input parsing.

The code in this PR could be used to refactor KokkosSparse_spadd.cpp, specifically it's main() function.
Ideally it would be replaced with the code below:

#define KOKKOSKERNELS_PERF_TEST_NAME spadd_perf_test
#include "KokkosKernels_perf_test_instantiation.hpp"

int main(int argc, char** argv) {
  return main_instantiation(argc, argv);
}

@lucbv lucbv added the AT: WIP label Feb 7, 2023
@lucbv lucbv self-assigned this Feb 7, 2023
@lucbv
Copy link
Contributor Author

lucbv commented Feb 7, 2023

@brian-kelley @e10harvey
This is still a work in progress, we can add/modify features proposed here, but eventually we could wrap a chunk of our performance test by using main_instantiation() to handle which execution space to run on.
The input parsing logic can be expanded and improved, for instance handling TPLs would be good!

} else if (0 == Test::string_compare_no_case(argv[argIdx], "--sycl")) {
params.use_sycl = atoi(argv[++argIdx]) + 1;
validated_input[argIdx] = 1;
}
Copy link
Contributor

@brian-kelley brian-kelley Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about instead of returning a flag for each arg index whether it's a common param, just return a vector of strings that has all the remaining, non-common args? That would make it easier for each perf test to parse and validate its own specific options.

This also needs bounds checking before reading argv[++argIdx] - for example if --cuda is the last argument, this will read past the end of the argv array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yeah, I missed the bounds checking issue in spadd again, but here we can fix it once and for all :)

@lucbv
Copy link
Contributor Author

lucbv commented Feb 8, 2023

Making a note that this is contributing to resolve issue #1674

@lucbv lucbv force-pushed the perf_test_wrapper branch from e63917a to 021716d Compare February 9, 2023 22:21
The goal of this work is to create a common core infrastructure
for the performance test in order to simplify maintenance.
Here two ideas are introduced:
1. the instantiation wrapper
2. the common input parser

both are trying to capture some of the implementation of our
performance test in generic functions that can be called instead
of duplicating logic around instantiation and command line input
parsing.

The new parsing routine checks the parameter name and that the associated
value can be casted properly. It also add some logic to remove
the arguments from argv and argc once they are parsed properly.
@lucbv lucbv force-pushed the perf_test_wrapper branch from f51e0ff to d3ffe82 Compare February 10, 2023 00:01
@lucbv lucbv added AT: RETEST Have this PR retested. and removed AT: WIP labels Feb 10, 2023
@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930

  • Build Num: 295
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

  • Build Num: 302
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020

  • Build Num: 216
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020_Light_LayoutRight

  • Build Num: 215
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC1020

  • Build Num: 178
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19

  • Build Num: 265
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 315
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG13CUDA10

  • Build Num: 235
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

  • Build Num: 130
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

  • Build Num: 128
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_ROCM520

  • Build Num: 124
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (lucbv/kokkos-kernels)
  • Branch: perf_test_wrapper
  • SHA: d3ffe82
  • Mode: TEST_REPO

Pull Request Author: lucbv

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_GCC930_Light_Tpls_GCC930

  • Build Num: 295
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CUDA11_CUDA11_LayoutRight

  • Build Num: 302
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020

  • Build Num: 216
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC1020_Light_LayoutRight

  • Build Num: 215
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC1020

  • Build Num: 178
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL19

  • Build Num: 265
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

  • Build Num: 315
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG13CUDA10

  • Build Num: 235
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_Tpls_ARMPL2110

  • Build Num: 130
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_A64FX_GCC1020

  • Build Num: 128
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_VEGA908_ROCM520

  • Build Num: 124
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
KOKKOSKERNELS_SOURCE_BRANCH perf_test_wrapper
KOKKOSKERNELS_SOURCE_REPO https://github.com/lucbv/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA d3ffe82
KOKKOSKERNELS_TARGET_BRANCH develop
KOKKOSKERNELS_TARGET_REPO https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA c2486ab
PR_LABELS AT: RETEST
PULLREQUESTNUM 1676
TEST_REPO_ALIAS KOKKOSKERNELS

@kokkos-devops-admin kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Feb 10, 2023
@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

@kokkos-devops-admin
Copy link

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@lucbv lucbv requested a review from brian-kelley February 10, 2023 17:38
@lucbv
Copy link
Contributor Author

lucbv commented Feb 10, 2023

One more review please : )

@kokkos-devops-admin
Copy link

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ brian-kelley ]!

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

Co-authored-by: brian-kelley <brian.honda11@gmail.com>
@lucbv lucbv merged commit 304bcda into kokkos:develop Feb 10, 2023
@lucbv lucbv deleted the perf_test_wrapper branch February 10, 2023 22:16
@kokkos-devops-admin
Copy link

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@kokkos-devops-admin
Copy link

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = None

<< "\t\t'--hip [deviceIndex]' |\n"
<< "\t\t'--sycl [deviceIndex]'\n\n"
<< "\tIf no parallel backend is requested, Serial will be used "
"(if enabled)\n\n";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding '--' to the end to denote the beginning of 'application' options.

num_threads = params.use_openmp;
} else if (params.use_threads) {
num_threads = params.use_threads;
}
Copy link
Contributor

@e10harvey e10harvey Feb 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error if both use_openmp and use_threads are set.

if (params.use_openmp) {
#if defined(KOKKOS_ENABLE_OPENMP)
std::cout << "Running on OpenMP backend.\n";
KOKKOSKERNELS_PERF_TEST_NAME<Kokkos::OpenMP>(argc, argv, params);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants