Fix GitHub CI failing on broken `develop` #1461

mzuzek · 2022-07-08T12:03:10Z

This PR addresses couple of issues that cause GitHub CI to fail with unrelated compilation errors (i.e. broken develop):

Kokkos updates:
- PR#5135 deprecates Kokkos::InitArguments replacing it with new Kokkos::InitializationSettings: here we update Kokkos initialization in performance tests accordingly;
  - Note: Deprecated num_numa was being set in perf_test/sparse/KokkosSparse_block_pcg.cpp.
- PR#5178 guards against private header inclusion: here we switch to Kokkos_StdAlgorithms.hpp for std algorithms and just drop other private includes (like Kokkos_Concepts.hpp);
- PR#5144 removes static from execution space print_configuration(): we call it on instances now;
Bugs:
- Bug in batched GESV (introduced by Add batched GESV #1384) and detected with Kokkos_ENABLE_DEBUG_BOUNDS_CHECK=ON introduced by Sparse and CI updates #1411 - see the details below;

kokkos-devops-admin · 2022-07-08T12:07:08Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

lucbv

Looks good to me, thanks for updating the tests @mzuzek

lucbv

Thanks for the additional clean-up

mzuzek · 2022-07-12T18:39:00Z

GESV bug details

@lucbv @kliegeois
In batched GESV (e.g. KokkosBatched_Gesv_Impl.hpp:553) 2D temp views are created with nxn+4 extents:

ScratchPadMatrixViewType tmp(member.team_scratch(0), n, n + 4);

Because input matrix layout was enforced on ScratchPadMatrixViewType (e.g. see lines 546-549), when the input was a subview created in dense unit tests, ScratchPadMatrixViewType had LayoutStride and n + 4 was passed by View constructor to LayoutStride constructor as a stride instead of second extent, like with regular layouts (I'd say it's a bit of a trap there in multidimensional Kokkos::View constructor...). Because in batched test tmp ended up being n x 1 rank-1 view instead of intended n x n+4 rank-2 view, when trying to split tmp into PDAD, PDY, D2, etc. with subview in KokkosBatched_Gesv_Impl.hpp:554-558 we got bounds violation errors in CI here:

[ RUN      ] serial.batched_scalar_team_gesv_static_pivoting_float
unknown file: Failure
C++ exception with description "Kokkos::subview bounds error ( Kokkos::ALL , 1 <= 3 - 0 )" thrown in the test body.
[  FAILED  ] serial.batched_scalar_team_gesv_static_pivoting_float (1 ms)

Solution

The fix proposed here allows tmp view to use default scratchpad layout - in hope it won't be LayoutStride - instead of imposing the one derived from input matrix.

It's hard for me to tell how this fix can affect the performance of GESV calculation based on PDAD, PDY, D2, etc. with regard to the underlying layout of these helper structures - so if that matters, please check that within your review, if possible.

mzuzek · 2022-07-12T18:41:05Z

@lucbv
Thank you for your reviews!
Hopefully, you can tolerate me piling up different fixes under one PR here... I can split them into separate PRs anytime, if you wish.

kliegeois · 2022-07-12T18:47:41Z

Thanks @mzuzek for having found the issue! I am wondering if we should revisit the template arguments of the GESV to allow the users to specify the layout of the temporary view.

lucbv

Yep, this should do it, thanks for all the fixes

lucbv · 2022-07-12T22:10:58Z

@mzuzek actually we need to keep all these fixes in a single PR otherwise the auto-tester would keep on failing since we test against the develop branch of Kokkos Core.

kokkos-devops-admin · 2022-07-13T18:07:29Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

kokkos-devops-admin · 2022-07-13T18:07:34Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed.

kokkos-devops-admin · 2022-07-13T18:11:47Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Build Num: 294
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Build Num: 287
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 1103
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 747
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 1091
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 1078
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 483
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (NexGenAnalytics/kokkos-kernels)

Pull Request Author: mzuzek

kokkos-devops-admin · 2022-07-13T19:46:23Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Build Num: 294
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Build Num: 287
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 1103
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 747
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 1091
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 1078
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 483
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	fix-kokkos-initargs
KOKKOSKERNELS_SOURCE_REPO	https://github.com/NexGenAnalytics/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`0060bcd`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`838ab1f`
PR_LABELS	AT: RETEST
PULLREQUESTNUM	1461
TEST_REPO_ALIAS	KOKKOSKERNELS

kokkos-devops-admin · 2022-07-13T19:46:41Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ lucbv ]!

kokkos-devops-admin · 2022-07-13T19:46:48Z

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

e10harvey · 2022-08-01T17:01:41Z

Thank you, @mzuzek !

Fix deprecated Kokkos::InitArguments

792a31b

mzuzek changed the title ~~Fix deprecated Kokkos::InitArguments~~ Fix deprecated Kokkos::InitArguments Jul 8, 2022

lucbv requested review from crtrott, e10harvey, brian-kelley and lucbv July 11, 2022 14:16

lucbv approved these changes Jul 11, 2022

View reviewed changes

lucbv added the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Jul 11, 2022

Mikołaj Zuzek added 2 commits July 11, 2022 20:36

Fix: don't use Kokkos private headers

01b14d2

Remove unused/duplicated headers

87b5723

lucbv approved these changes Jul 11, 2022

View reviewed changes

mzuzek changed the title ~~Fix deprecated Kokkos::InitArguments~~ Fix CI compilation errors caused by recent Kokkos develop updates Jul 11, 2022

Fix: Use default layout for temp views in batched GESV

8e6986d

mzuzek changed the title ~~Fix CI compilation errors caused by recent Kokkos develop updates~~ Fix few items breaking GitHub CI Jul 12, 2022

mzuzek changed the title ~~Fix few items breaking GitHub CI~~ Fix few things breaking GitHub CI Jul 12, 2022

mzuzek changed the title ~~Fix few things breaking GitHub CI~~ Fix GitHub CI failing on broken develop Jul 12, 2022

lucbv self-requested a review July 12, 2022 18:13

lucbv added AT: PRE-TEST INSPECTED Mark this PR as approved for testing. and removed AT: PRE-TEST INSPECTED Mark this PR as approved for testing. labels Jul 12, 2022

mzuzek mentioned this pull request Jul 12, 2022

Missing error check for some View constructors kokkos/kokkos#5205

Closed

Fix: call print_configuration() on instances (no longer static)

0060bcd

lucbv approved these changes Jul 12, 2022

View reviewed changes

lucbv added AT: PRE-TEST INSPECTED Mark this PR as approved for testing. AT: RETEST Have this PR retested. and removed AT: PRE-TEST INSPECTED Mark this PR as approved for testing. labels Jul 12, 2022

brian-kelley mentioned this pull request Jul 12, 2022

trsv: remove assumptions about entry order within rows #1463

Merged

kokkos-devops-admin removed the AT: PRE-TEST INSPECTED Mark this PR as approved for testing. label Jul 13, 2022

kokkos-devops-admin removed the AT: RETEST Have this PR retested. label Jul 13, 2022

lucbv merged commit 979cffa into kokkos:develop Jul 13, 2022

ndellingwood mentioned this pull request Jul 13, 2022

update nightly testing scripts #1462

Merged

mzuzek deleted the fix-kokkos-initargs branch July 14, 2022 09:53

This was referenced Jul 14, 2022

Fix incorrect function call in KokkosBatched::TeamGEMV unit test #1444

Merged

Move Set/Scale unit test to KokkosBlas #1455

Merged

ndellingwood added the CompatibilityUpdate label Jul 28, 2022

mzuzek mentioned this pull request Sep 29, 2022

task1: implementations NexGenAnalytics/kokkos-kernels#2

Open

Fix GitHub CI failing on broken develop #1461

Fix GitHub CI failing on broken develop #1461

Conversation

mzuzek commented Jul 8, 2022 • edited Loading

kokkos-devops-admin commented Jul 8, 2022

lucbv left a comment

Choose a reason for hiding this comment

lucbv left a comment

Choose a reason for hiding this comment

mzuzek commented Jul 12, 2022

GESV bug details

Solution

mzuzek commented Jul 12, 2022

kliegeois commented Jul 12, 2022

lucbv left a comment

Choose a reason for hiding this comment

lucbv commented Jul 12, 2022

kokkos-devops-admin commented Jul 13, 2022

kokkos-devops-admin commented Jul 13, 2022

kokkos-devops-admin commented Jul 13, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Jenkins Parameters

Using Repos:

kokkos-devops-admin commented Jul 13, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Jenkins Parameters

kokkos-devops-admin commented Jul 13, 2022

kokkos-devops-admin commented Jul 13, 2022

e10harvey commented Aug 1, 2022

Fix GitHub CI failing on broken `develop` #1461

Fix GitHub CI failing on broken `develop` #1461

mzuzek commented Jul 8, 2022 •

edited

Loading