EPIC: Issues with Trilinos PR testing 2022-08

[bartlettroscoe]

There are a number of issues impacting Trilinos PR testing currently that are blocking many active PRs from passing the PR tester:

• Random mass test failures showing "Error initializing RM connection. Exiting" on 'vortex' #6861
• Thyra: EpetraOperatorWrapper_UnitTests.cpp cannot open "Trilinos_Util_CrsMatrixGallery.h" blocking all PRs changing Tpetra #10842
• Framework: Mysterious 6 failures in package Zoltan2Sphynx with no data on CDash going back to at least 2022-05-18 #10836
• TpetraTSQR_SequentialTsqr_[non]contiguousCacheBlocks_MPI_1 tests failing randomly with 'illegal instruction was encountered' in PR rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release builds starting 2022-07-20 #10847 ... Tests have been disabled for now
• INVALID: Framework: PR build showing missing test executable but no build failure #10841
• Framework: trilinos-cdash.sandia.gov appears to be down 2022-08-10 #10857
• Framework: PR builds failing due to "No space left on device" 2022-08 #10875
• Random failures due to jumbled output in TpetraCore_Bug7745_MPI_4 and TpetraCore_MultiVector_LocalViewTests_MPI_4 starting 2022-08-05? #10885
• New asici0xy nodes brought online 2022-08-16 in PR testing missing system-installed BLAS and LAPACK #10893
• Random failures due to jumbled output in TpetraCore_MatrixMarket_Tpetra_CrsMatrix_Dist_Binary_simple_MPI_1 breaking PR builds starting 2022-07-08 #10898
• Framework: Jenkins PR driver down and skipping of odd-numbered PRs starting 2022-08-16 #10917
• Pliris: Pliris tests failing in PR ATS2 CUDA build which should not even be running #10931... Pliris is disabled in ATS2 CUDA PR builds with merge of PR #10930

Also of interest:

• Framework: Some PR build errors showing up as strange 'cmake --build . --config Release -- -j29 -k 0' errors #10823
• Improve instructions for reproducing PR builds using GenConfig system #10867
• Framework: PR autotester is incorrectly testing against old version of 'develop' #10840
• Framework: Tracking excessive numbers of PR testing retries and failures #10849
• Framework: PR builds taking upwards of a day to run since 2022-08-01 #10899

[csiefer2]

#10896

[bartlettroscoe]

#10896

As explained in #10896 (comment), there are not causes of PR failures, they are victims of a build failure that was also reported in those same PR builds (or not for the clang-10.0.0 builds using older CMake 3.17.1 as per #10893 (comment) :-( ).

[bartlettroscoe]

Even with all of the fixes in the linked issues above, we still have a large log jam in PR testing. For example, right now this issues query shows there are 8 PRs that have the AT: RETEST label on them. It looks like something is wrong with the PR tester is that only one set of PR build is running at one time. The outer driver for the second set of PRs looks to be broken (see https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/). I will post a Trilinos HelpDesk issue.

Update: Here is the helpdesk issue: TRILINOSHD-186

[bartlettroscoe]

FYI: I posted TRILINOSHD-187 to see if they can turn off auto-retest of failing PR builds. That is often a waste of computing resources as the developer has not even had a chance to look over the failed PR build results to see if rerunning the PR builds is likely to fix anything.

[bartlettroscoe]

As noted above, only one of two Jenkins Trilinos_autotester_driver_inst instances is running and there seems to be logic built into assume that the other instance is running PR builds. For example:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_0/4489/consoleFull

shows:

22:00:38 PROCESSING prnum = 10897
22:00:38    PROCESSING PULL REQ# 10897
22:00:38       - PRINFO - Mergeable: True; State: open; Title: Improve TrilinosInstallTests to make more clear (#10896); By: bartlettroscoe
22:00:38       - SOURCE - Repo: bartlettroscoe/Trilinos; branch: 10896-improve-doc-install-tests; sha: 00ca3870b23ddea877be9f1abbaf1e3bfee833ad; User: bartlettroscoe
22:00:38       - TARGET - Repo: trilinos/Trilinos; branch: develop; sha: f4b73d31ef4f77626c1c7c8c9a86034cc64fa011; User: trilinos
22:00:38       - PR From Fork = True
22:00:38 *** NOTICE: This PR #10897 is not tested by this instance of the pullrequestautotester_inst_0; Instance 1 will test it - SKIPPING THIS PULL 

So the PR tester logic seems to think that the other instance (Instance 1) will be testing my PR #10897 but that instance appears to be broken a shown at, for example:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/3857/console

showing:

21:42:00 Started by timer
21:42:00 Running as SYSTEM
21:42:01 Building remotely on [pr-host-1](https://do.sandia.gov/trilinos-ci/computer/pr-host-1) (ceecloud_PR_host_1 PR_host) in workspace <...>/Trilinos_autotester_driver_inst_1
21:42:01 [WS-CLEANUP] Deleting project workspace...
21:42:01 [WS-CLEANUP] Deferred wipeout is used...
21:42:07 ERROR: [WS-CLEANUP] Cannot delete workspace: Remote call on pr-host-1 failed
21:42:07 ERROR: Cannot delete workspace: Remote call on pr-host-1 failed
21:42:07 Finished: FAILURE

I think as a result, all of the 8 PRs with AT: RETEST put on them seem to be assigned to run in PR "Instance 1" which is broken.

As a result, no PR builds have started since Aug 19, 2022 - 18:08 MDT shown here.

This system does not seem to be very robust to instances of these drivers going down.

[bartlettroscoe]

Looking over the PR builds being skipped with the message:

*** NOTICE: This PR #<prnum> is not tested by this instance of the pullrequestautotester_inst_0; Instance 1 will test it - SKIPPING THIS PULL REQUEST

at:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_0/4489/consoleFull

I am noticing that all of the <prnum> like #8161, #8737, #8855, ..., #10909 (41 PRs in total) are all odd numbers. That suggests that the PR tester's approach to load balancing is to use <prnum> % num_pr_drivers to compute which PR drivers instance 0, 1, ... will be testing a given PR. So with two running instances, odd numbered PRs compute to <prnum> % num_pr_drivers.

So if you want to get your PR to be tested, just open a new PR and make sure the PR number is even :-)

[bartlettroscoe]

Looking at the history for the broken PR tester driver:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/

it seems that the Trilinos_autotester_driver_inst_1 last ran successfully on 'Aug 16, 2022, 2:03:02 PM'.

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/3724/

After that, starting on 'Aug 17, 2022, 6:37:29 PM' for:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/3725/

that instance was broken and not running any PR builds.

So it seems that the PR tester implementation for the last 4 days has been running only one set of PR builds at a time and then only testing even-numbered PRs :-)

And to back this up, this CDash query shows that the last time an odd-numbered PR was tested was Aug 17, 2022 - 14:18 MDT with the build:

• PR-10851-test-ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables-928

After that, only even-numbered PRs were tested.

And if you look at the current list of PRs with AT: RETEST on them here, you will see they are all odd-numbered PRs.

(NOTE: Those dates don't exactly match up so not sure how an odd-numbered PR was able to be tested after 'Aug 16, 2022, 2:03:02 PM' but the general trend of only even-number PRs getting tested and only odd-numbered PRs that still have AT: RETEST holds.)

[bartlettroscoe]

FYI: I created the even-numbered PR #10912 to test the above even-only PR testing hypothesis. But it looks like PR builds will not start up again until tomorrow morning after 7 AM MDT so we will have to wait until then.

[bartlettroscoe]

As predicted above, the next job fired off at 7:16 AM MDT with:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_0/4490/

and it is running PR builds for the even-numbered PR #10912 showing:

09:22:10    * ========================
09:22:10    * PERFORMING TESTING ON PULL REQ# 10912 (Run Number 1/2)
09:22:10    * ========================

(Jenkins must be showing times according to my browser so this must be 09:22:10 EDT, not MDT).

[bartlettroscoe]

Another problem with the PR builds:

• Pliris: Pliris tests failing in PR ATS2 CUDA build which should not even be running #10931

[bartlettroscoe]

FYI: I think the major errors blocking PR builds from passing have been resolved. Now there are just a few random failures like #6861 that are still taking down PR testing iterations.

[bartlettroscoe]

FYI As (not) shown in this GitHub Issue Query there are no more PRs with the AT: RETEST label. And no PR builds have started in the last 22 hours And the two drivers are running with the latest runs shown at:

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_0/4661/consoleFull

and

• https://do.sandia.gov/trilinos-ci/job/Trilinos_autotester_driver_inst_1/3987/consoleFull

And this GitHub Issue Query shows that there are just 8 open PRs with approved reviews and of those 7 have a failing PR builds status.

Therefore, the logjam of PR builds is over. Yes, there are still a few sources of random failures that may trigger a failing PR testing iteration (i.e. #6861 and #10989) but everything else seems to have been addressed.

I will now close this EPIC and remove the pin.