Test Zoltan2_OrderingScotch_MPI_4 timed out in CI testing on 3/16/2018

[bartlettroscoe]

@trilinos/zoltan2, @trilinos/framework

Description

The test Zoltan2_OrderingScotch_MPI_4 timed out at 10 minutes in the first CI build Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI of the morning as shown at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3372395
• https://testing-vm.sandia.gov/cdash/testDetails.php?test=44307470&build=3372508

which shows:

Starting everything
UserInputForTests, Read: ./simple_ordering.mtx
UserInputForTests, Read: ./simple_ordering_coord.mtx
NumRows     = 25
NumNonzeros = 25
NumProcs = 4
Ordering does not support distributed matrices.
Ordering does not support distributed matrices.
Going to solve
Ordering does not support distributed matrices.

It looks like one of the 4 nodes may be hanging. This error look identical to the still-open issue #2131 where that test should have been disabled from the standard CI build on 1/8/2018.

The test is also failing in the other build Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI_AAOP that enables Scotch:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-16&filtercombine=and&filtercount=2&showfilters=1&filtercombine=and&field1=testname&compare1=61&value1=Zoltan2_OrderingScotch_MPI_4&field2=status&compare2=62&value2=Passed

However, there are no changes pulled in the CI iteration shown at:

• https://testing-vm.sandia.gov/cdash/viewNotes.php?buildid=3372508#!#note1

that would seem to account for this failure or that would have re-enabled this test.

So how did this test get re-enabled?

Related Issues

• Related issues: Test Zoltan2_OrderingScotch_MPI_4 hanging on crf450 #2131, Add sems-scotch/6.0.3/nopthread_64bit_parallel TPL to automated PR builds of Trilinos #2065

[bartlettroscoe]

It looks like the only place that this test is run is the CI builds that I set up. It looks like these test is not being run in the auto PR builds, or it would show up in the query:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-16&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=1&showfilters=1&field1=testname&compare1=61&value1=Zoltan2_OrderingScotch_MPI_4

[MicheldeMessieres]

@bartlettroscoe I was testing this on the SEMS machine - but I thought I had just done local changes. I'm checking now but did not do any commit I was aware of.

[bartlettroscoe]

Looking at the query:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=3&showfilters=1&filtercombine=and&field1=testname&compare1=61&value1=Zoltan2_OrderingScotch_MPI_4&field2=buildname&compare2=61&value2=Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI&field3=buildstarttime&compare3=84&value3=now

it looks like this test never got disabled in the post-push CI build. Instead, looking at commit 5c01d87, it only got disabled in the checkin-test-sems.sh script.

Therefore, it looks like we need to disable this test in the post-push CI build as well.

But I think that unless Scotch gets enabled in the auto PR builds, then we should disable Scotch in this CI build since people are not using the checkin-test-sems.sh script to push as much now because they are being told to use the auto PR build.

[MicheldeMessieres]

@bartlettroscoe I just recently got back up and running on that SEMS machine. Using ssh protocol you recommended worked for git and proxy exports @kddevin gave me restored outside access. We identified the hang issue (related to handling of 64 bit scotch) for that test and resolution should be coming soon.

[bartlettroscoe]

I was testing this on the SEMS machine - but I thought I had just done local changes. I'm checking now but did not do any commit I was aware of.

@MicheldeMessieres, I don't know that I can explain why this latest set of builds show this failure since the prior CI iteration that tested Zoltan2 as part of Tpetra changes shown at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3370648&

passed. Those are the only changes to Trilinos in the last 24 hours that would explain this.

Very strange.

But given that the GCC 7.2.0 build this morning showed this test passing at:

• https://testing-vm.sandia.gov/cdash/testDetails.php?test=43121270&build=3372455

shows that this problem does not exhibit itself with all compilers on all machines.

[bartlettroscoe]

I pushed the commit 270c3cd which disables this test in the CI build (and therefore all Trilinos automated builds it would seem). But this test will still get enabled and run in all other builds of Trilinos that enable the Scotch TPL.

[kddevin]

Thank you for reporting this issue. It is now fixed in Trilinos/develop with PR #2415.
When you re-enable the test, please let us know if you see further problems.

[bartlettroscoe]

The test Zoltan2_OrderingScotch_MPI_4 was shown to be newly added and passing in the CI iteration this morning shown at:

• https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3381445

(see the +1 superscript by 101 passing Zoltan2 tests) and shown more explicitly at:

• https://testing-vm.sandia.gov/cdash/viewTest.php?onlydelta&buildid=3381534

This issue looks resolved. Closing as complete.