Initial adjustment of Primary Tested set of Trilinos Packages and TPLs

[bartlettroscoe]

Next Action Status:

Sent email to SNL customers of Trilinos, asking for list of Trilinos packages and TPLs. Next: Wait for responses ...

CC: @jwillenbring, @bmpersc, @maherou, @trilinos/framework

Blocked By: #158

Blocking: #370

Description:

This story is to do an initial adjustment (mostly an extension) the set of Primary Tested (PT) TPLs to include HDF5, Netcdf, Boost, SWIG and any others needed to fully build and test packages like SEACAS, STK, etc. This initial set of packages and TPLs will be limited to what is provided by the SEMS Dev Env (see #158). Otherwise, some packages that could be included based on that set of TPLs should otherwise be excluded from the set of PT packages as guided by the needs of important Trilinos customers and critical research efforts (which are need to be well protected). Purely exploratory software or packages without important current users uses should generally be excluded from the PT set (to speed up the builds and tests). Also, non-critical configurations (e.g. complex, float) should be avoided to speed up builds and testing.

This initial story will be to support the usage of the checkin-test.py script to update from the 'develop' to the 'master' branch (see #370).

Tasks:

1. Get initial feedback from Trilinos developers and stakeholders (captured in emails to Ross) [Done]
2. Document, analyze and combine the lists from SNL customers (see TRIL-59) ...
3. Come up with a new list of PT Packages and TPLs (that can be supported by the SEMS Dev Env) ...
4. Fill out the "Testing Category" column in the Trilinos Package Classification Database ...
5. Update the file Trilinos/PackagesList.cmake and TPLsList.cmake

[bartlettroscoe]

I just sent an email to internal SNL customers of Trilinos in order to try to gather up the list of important Trilinos packages. Unfortunately, I can't include those emails here because they name specific SNL codes. But I will gather up the final list in this Issue ticket.

[bartlettroscoe]

NOTE: I just created the internal SON ticket:

• https://software-sandbox.sandia.gov/jira/browse/TRIL-59

to process the list of "important" packages and TPLs for internal SNL customers. The outcome from that story will be an updated candidate list of PT packages and TPLs.

[bartlettroscoe]

I completed the analysis of the Trilinos packages, SE package, and TPLs used by a bunch of Trilinos customers and created a combined list from all of these. You can see what customers were considered and where the data came from by looking at:

• https://software-sandbox.sandia.gov/jira/browse/TRIL-59

The combined list sorted alphabetically is given below. The challenge that we face is that large set of TPLs that all of our Trilinos customers are using:

• AMD BLACS BLAS Boost BoostLib DLlib HDF5 LAPACK METIS MPI MUMPS Netcdf PARDISO_MKL ParMETIS PETSC Pthread QT SCALAPACK SuperLU SuperLUDist UMFPACK X11 y12m Zlib

The problem is that the SEMS env can only provide the TPLs:

• Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib

What is missing from SEMS is:

• AMD, BLASC, MUMPS, PARDISO_MKL, PETSC, QT, SCALAPACK, SuperLUDist, UMFPACK, y12m

Not much we can do about that if we are going to build our CI system on the SEMS env. It has what it has.

I am going to put this story into review now and move on with #482.

@trilinos/framework, any comments on this?

---

Full set of packages:

Amesos Amesos2 Anasazi AztecOO Belos Claps Domi Drekar DrekarMHD Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos Mesquite ML MOOCHO MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2

Full set of SE packages:

Amesos Amesos2 Anasazi AztecOO Belos Claps Domi Drekar DrekarMHD Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 IntrepidCore Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore KokkosTPL Mesquite ML MOOCHO MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASBlot SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASFastq SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASPLT SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASTxtexo Shards ShyLU ShyLUCore STK STKClassic STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2

Full set of TPLs:

AMD BLACS BLAS Boost BoostLib DLlib HDF5 LAPACK METIS MPI MUMPS Netcdf PARDISO_MKL ParMETIS PETSC Pthread QT SCALAPACK SuperLU SuperLUDist UMFPACK X11 y12m Zlib

[srajama1]

I see Scotch is missing.

[bartlettroscoe]

I see Scotch is missing.

That is interesting. Looks like none of the internal Trilinos customers that responded to my request are using Scotch with Trilinos. You can see the full set of customers that were considered in:

• https://software-sandbox.sandia.gov/jira/browse/TRIL-59

But also note that the SEMS env only provides a 32-bit build for Scotch which is in-compatible with the 64-bit version they provide for ParMETIS. You can read about that in #475.

[srajama1]

Scotch is a recent addition. SIERRA officially started testing it late September, if my memory is right.

[bartlettroscoe]

Scotch is a recent addition. SIERRA officially started testing it late September, if my memory is right.

Good to know, thanks. I did a search of the SEMS JIRA project and I found:

• https://sems.sandia.gov/jira/browse/SEMS-1108

that suggests that SEMS might now provide a compatible 64-bit version of Scotch. Can anyone confirm that? @kddevin, have you been able to test out the SEMS env with 64-bit ParMETIS and Scotch both enabled with Zoltan and Zoltan2?

[srajama1]

Is there a 32-bit version. Tacho works with 32-bit. We can add a 64-bit version interface as well.
@kyungjoo-kim

[bartlettroscoe]

Is there a 32-bit version. Tacho works with 32-bit. We can add a 64-bit version interface as well.

The problem is that you can't enabled both the 64-bit version of ParMETIS and the 32-bit version of Scotch at the same time. If you do, a bunch of Zoltan and Zoltan2 tests fail. That is no good.

[kyungjoo-kim]

Tacho does not require for 32 bit version specifically. Does SIERRA use a 32 bit version ?

[kddevin]

1. Zoltan2 works fine with mixed 32-bit/64-bit TPL index types. Only Zoltan requires consistent ID sizes among the TPLs; it is unlikely we will ever have resources to remove that constraint.
2. All Zoltan and Zoltan2 tests pass with SEMS' 64-bit ParMETIS and Scotch, and with SEMS' 32-bit ParMETIS and Scotch.

[bartlettroscoe]

Zoltan2 works fine with mixed 32-bit/64-bit TPL index types. Only Zoltan requires consistent ID sizes among the TPLs; it is unlikely we will ever have resources to remove that constraint.

All Zoltan and Zoltan2 tests pass with SEMS' 64-bit ParMETIS and Scotch, and with SEMS' 32-bit ParMETIS and Scotch.

@kddevin, has anyone tried this with the ParMETIS and Scotch TPLs provided by the SEMS Env? If not, we can try this later once we get #482 done.

Adding more TPLs to our CI and Nightly testing is something that we will need to do gradually as we can. (We just need infrastructure manpower to help with this). If you look at the list of TPLs that are actually used by our important Sandia Trilinos customers listed above, you will see that we are falling far short with what Trilinos is currently testing in Nightly builds and what is provided by the SEMS env.

[kddevin]

@bartlettroscoe I don't understand. What is "this" in your question above?

[bartlettroscoe]

What is "this" in your question above?

Sorry, I should have been more explicit. Has any Zoltan or Zoltan2 developer tried the updated SEMS-provided 64-bit Scotch and ParMETIS TPLs? It looks like these are now present under:

• /projects/sems/install/rhel6-x86_64/sems/tpl/scotch/6.0.3/gcc/4.8.4/openmpi/1.8.7/nopthread_64bit_parallel/
• /projects/sems/install/rhel6-x86_64/sems/tpl/parmetis/4.0.3/gcc/4.8.4/openmpi/1.8.7/64bit_parallel/

If not, that is fine. I was just asking.

[kddevin]

Yes; see my comment 2 above. All tested. All good. Thanks.

[bartlettroscoe]

Realized that we need to make a few changes to the list of packages and SE packages listed above.

First, Claps is an EX. It is not actively developed in Trilinos and there is no reason to test it.

Second, I accidentally listed Drekar and DrekarMHD. Those are not even Trilinos packages.

Third, TriKota depends on Dakota. I don't think we should be forcing Trilinos developers to have to build Dakota when the test Trilinos. I think that is asking too much. Also, there is not a solid integration model for what version of Dakota is compatible with TriKota. Therefore, I think it makes sense to just let Albany test TriKota.

Forth, Mesquite and MOOCHO are in a separate git repo from the main Trilinos repo. Therefore, it makes no sense to include it in the set of PT packages.

So taking out Claps, Drekar, DrekarMHD, Mesquite, and TriKota, the new list of primary tested packages and TPLs becomes:

• New set of PT packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos ML MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT SE packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 IntrepidCore Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore KokkosTPL ML MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASBlot SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASFastq SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASPLT SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASTxtexo Shards ShyLU ShyLUCore STK STKClassic STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT TPLs: Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib

[bartlettroscoe]

Another adjustment needs to be made. The subpackage IntrepidCore no longer exists in Trilinos. It was taken out in the commit:

commit 889d5e8e493cafe3ef151d6b886e815053e704a5
Author: avdelor <avdelor@sandia.gov>
Date:   Tue Jan 5 16:50:14 2016 -0700

    Intrepid:

    fixed phalanx dependencies
    took intrepid out of core subpackage

diff --git a/packages/intrepid/cmake/Dependencies.cmake b/packages/intrepid/cmake/Dependencies.cmake
index f1b64a3..c2eba4a 100644
--- a/packages/intrepid/cmake/Dependencies.cmake
+++ b/packages/intrepid/cmake/Dependencies.cmake
@@ -1,8 +1,3 @@
-SET(SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS
-  #SubPackageName       Directory         Class    Req/Opt
-  #
-  Core                  core               PT      OPTIONAL
-  )
 SET(LIB_REQUIRED_DEP_PACKAGES Teuchos Shards Sacado)
 SET(LIB_OPTIONAL_DEP_PACKAGES KokkosCore KokkosAlgorithms)
 SET(TEST_REQUIRED_DEP_PACKAGES)

CASL VERA is using very old version of Intrepid before this commit was made.

Removing IntrepidCore the new set of of PT package are:

• New set of PT packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos ML MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT SE packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore KokkosTPL ML MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASBlot SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASFastq SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASPLT SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASTxtexo Shards ShyLU ShyLUCore STK STKClassic STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT TPLs: Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib

[bartlettroscoe]

Another change to the list of SE packages. Seems that Kokkos no longer has a subpackage called "KokkosTPL". It was removed in this commit:

commit e1e44072a7e8c1c76037f4c7766f321062b2dd6c
Author: Carter Edwards <hcedwar@sandia.gov>
Date:   Tue Apr 7 10:45:15 2015 -0600

    Kokkos: Remove unused TPL subdirectory & subpackage.

D       packages/kokkos/TPL/CMakeLists.txt
M       packages/kokkos/config/configure_tpetra_kokkos_cuda_nvcc_wrapper.sh
M       packages/kokkos/core/cmake/Dependencies.cmake

Only one Trilinos customer listed that (you can see which one by examining the combined_list_of_trlinos_customer_enables.sh script in TRIL-59). That customer must have been using a very old version of Trilinos when they reported this.

The update list of enables removing KokkosTPL is:

• New set of PT packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos ML MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT SE packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore ML MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASBlot SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASFastq SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASPLT SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASTxtexo Shards ShyLU ShyLUCore STK STKClassic STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT TPLs: Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib

[bartlettroscoe]

I went back and put the script combined_list_of_trlinos_customer_enables.sh under version control and carefully make all of the tweaks to it that are described above and few others (see TRIL-59). Now I can get a local configure of Trilinos to exactly match the updated list and it looks right.

Here is the updated list of PT enables:

• New set of PT packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos ML MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT SE packages: Amesos Amesos2 Anasazi AztecOO Belos Domi Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore ML MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASTxtexo Shards ShyLU ShyLUCore STK STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TrilinosCouplings Triutils Xpetra Zoltan Zoltan2
• New set of PT TPLs: Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib

[bartlettroscoe]

This is complete and is being used in new CI build. It is unlikely that anyone will seriously review this so closing as complete.

Also, we are no longer going to purse extending the checkin-test-sems.sh (or checkin-test.py) script for Trilinos (see #482 (comment)). No idea what selection of packages and TPLs will be used for new automatic PR system but that can be determined later by someone else.