Ifpack2 tests broken in standrd CI build starting 8/11/2017

[bartlettroscoe]

CC: @trilinos/framework, @trilinos/intrepid2, @kyungjoo-kim

Description:

The package Intrepid2 is now showing a build failure in the CI build with the first breaking CI iteration yesterday shown at:

• https://testing.sandia.gov/cdash/index.php?project=Trilinos&parentid=3050255

The build failures shown at:

• https://testing.sandia.gov/cdash/viewBuildError.php?buildid=3050256

show:

/scratch/rabartl/Trilinos.base/SEMSCIBuild/Trilinos/packages/intrepid2/refactor/unit-test/Orientation/Serial/test_orientationtools_coeff_matrix.cpp:54:54: fatal error: test_orientationtools_tet_coeff_matrix.hpp: No such file or directory
 #include "test_orientationtools_tet_coeff_matrix.hpp"
                                                      ^
compilation terminated.

Looking at the new commits pulled for this CI iteration at:

• https://testing.sandia.gov/cdash/viewNotes.php?buildid=3050256##note0

we see that the following commits have caused this break:

70c425a:  Intrepid2 - tested with triangle hcurl and hdiv
Author: Kyungjoo Kim <kyukim@sandia.gov>
Date:   Fri Aug 11 15:18:41 2017 -0600

M	packages/intrepid2/refactor/src/Orientation/Intrepid2_OrientationTools.hpp
M	packages/intrepid2/refactor/src/Orientation/Intrepid2_OrientationToolsDefMatrixData.hpp
M	packages/intrepid2/refactor/unit-test/Orientation/Serial/test_orientationtools_coeff_matrix.cpp
M	packages/intrepid2/refactor/unit-test/Orientation/test_orientationtools_hex_coeff_matrix.hpp
M	packages/intrepid2/refactor/unit-test/Orientation/test_orientationtools_quad_coeff_matrix.hpp
M	packages/intrepid2/refactor/unit-test/Orientation/test_orientationtools_tri_coeff_matrix.hpp

d9897aa:  Intrepid2 - hcurl and hdiv of triangle and tet always requires orientation
Author: Kyungjoo Kim <kyukim@sandia.gov>
Date:   Fri Aug 11 15:17:24 2017 -0600

M	packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_HCURL_TET_In_FEM.hpp
M	packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_HCURL_TET_In_FEMDef.hpp
M	packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_HCURL_TRI_In_FEM.hpp
M	packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_HDIV_TET_In_FEM.hpp
M	packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_HDIV_TRI_In_FEM.hpp

98561e5:  Intrepid2 - triangle hcurl and hdiv orientation setup tested
Author: Kyungjoo Kim <kyukim@sandia.gov>
Date:   Fri Aug 11 13:38:24 2017 -0600

M	packages/intrepid2/refactor/src/Orientation/Intrepid2_OrientationToolsDefCoeffMatrix_HCURL.hpp
M	packages/intrepid2/refactor/src/Orientation/Intrepid2_OrientationToolsDefCoeffMatrix_HDIV.hpp
M	packages/intrepid2/refactor/unit-test/Orientation/Serial/test_orientationtools_coeff_matrix.cpp
M	packages/intrepid2/refactor/unit-test/Orientation/test_orientationtools_tri_coeff_matrix.hpp

Looking at the push log for Trilinos we see the push:

Fri Aug 11 15:20:17 MDT 2017

commit 70c425ab810664a6470bc02d4e0c58ed15b130c4
Author:     Kyungjoo Kim <kyukim@sandia.gov>
AuthorDate: Fri Aug 11 15:18:41 2017 -0600
Commit:     Kyungjoo Kim <kyukim@sandia.gov>
CommitDate: Fri Aug 11 15:18:41 2017 -0600

    Intrepid2 - tested with triangle hcurl and hdiv

Commits pushed:
70c425a Intrepid2 - tested with triangle hcurl and hdiv
d9897aa Intrepid2 - hcurl and hdiv of triangle and tet always requires orientation
98561e5 Intrepid2 - triangle hcurl and hdiv orientation setup tested

As you can see from that push log, there is no evidence that the checkin-test-sems.sh script was used to test and push this change.

This is just a test failure, and not a library failure. However, anyone trying to use the checkin-test-sems.sh script to push changes to any of the following packages upstream from Ifpack2 will have their push stopped due to this failing test build:

• Kokkos Teuchos KokkosKernels RTOp Sacado Epetra Shards Triutils Tpetra TrilinosSS EpetraExt Thyra Xpetra Galeri Amesos Pamgen

Therefore, this needs to be fixed ASAP or this test needs to be disabled so that it does not stop other people's pushes.

[bartlettroscoe]

I am going to update TriBITS to allow the disable of just a single executable and then externally disable this test build for just the CI build. That way, this failure will not trip up anyone trying to use the checkin-test-sems.sh script to push to:

• Kokkos Teuchos KokkosKernels RTOp Sacado Epetra Shards Triutils Tpetra TrilinosSS EpetraExt Thyra Xpetra Galeri Amesos

Then the Intrepid2 developers can fix this failing test in their leisure.

[bartlettroscoe]

I just pushed the commit 00bca64:

commit 00bca6466f45588d0df3311b12ce795162259e7b
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Sat Aug 12 09:50:46 2017 -0600

    Disable build and run of failing Intrepid2 test OrientationToolsCoeffMatrix (#1600)
    
    This disables the build and running of the test:
    
      Intrepid2_refactor_unit-test_Orientation_Serial_Test_OrientationToolsCoeffMatrix
    
    This is needed to get the CI build working again (see #1600, #1304).
    
    Build/Test Cases Summary
    Enabled Packages: Intrepid2
    Disabled Packages: PyTrilinos,Claps,TriKota
    0) MPI_RELEASE_DEBUG_SHARED_PT => passed: passed=82,notpassed=0 (3.13 min)
    Other local commits for this build/test group: 3a3bde5, 479735d

M       cmake/std/BasicCiTestingSettings.cmake

This only impacts the standard CI build. The test will still be enabled for all local builds and other nightly builds on CDash.

To revert this disable and then reproduce and fix the issue, please do:

$ cd Trilinos/
$ git checkout develop
$ git pull
$ git revert 00bca6466f45588d0df3311b12ce795162259e7b

Then do the standard CI build for Intrepid2 using:

$ cd CHECKIN/
$ ./checkin-test-sems.sh --enable-all-packages=off --no-enable-fwd-packages --enable-packages=Intrepid2 --local-do-all

Then one can iterative reconfigure, build, run tests, etc. as described at:

• https://github.com/trilinos/Trilinos/wiki/Policies-%7C-Safe-Checkin-Testing#reproducing-failures-manually

Then, once new commit(s) have been made that fix the issue, please test and push with:

$ cd Trilinos/
$ cd CHECKIN/
$ ./checkin-test-sems.sh --do-all --push

If you have questions about this please contact me and @trilinos/framework

[kyungjoo-kim]

I am sorry that I did not use checkin test and did not find out the missing file in the commit. It is fixed now.

      1 commit 76214f57e01e00610a565e735c969c55f0c7d8a7
      2 Author: Kyungjoo Kim <kyukim@sandia.gov>
      3 Date:   Sun Aug 13 23:32:00 2017 -0600
      4 
      5     Intrepid2 - dummy commit to checkin test
      6     
      7     Build/Test Cases Summary
      8     Enabled Packages: Intrepid2
      9     Disabled Packages: PyTrilinos,Claps,TriKota
     10     Enabled all Forward Packages
     11     0) MPI_RELEASE_DEBUG_SHARED_PT => passed: passed=512,notpassed=0 (10.90 min)

[william76]

This is also affecting the clean and nightly builds:

• https://testing.sandia.gov/cdash/index.php?project=Trilinos&date=2017-08-14

As I was typing this I saw that @kyungjoo-kim has put in a fix.

[bartlettroscoe]

@kyungjoo-kim,

You forgot to revert the disable commit as instructed above:

$ cd Trilinos/
$ git checkout develop
$ git pull
$ git revert 00bca6466f45588d0df3311b12ce795162259e7b

That will turn the test on but we need to make sure it is passing before pushing.