ats2 xl build failing starting 2021-02-10

[e10harvey]

CC: @trilinos/framework, @jwillenbring (Trilinos Framework Product Lead), @bartlettroscoe

Next Action Status

Description

As shown in this query the builds:

• Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt
• Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_dbg

started failing on testing day 2021-02-10.

All of the builds are show the following configure output like shown here:

CMake Error at /path/to/tpls/cmake/cmake-3.18.0/share/cmake/Modules/CMakeTestCXXCompiler.cmake:59 (message):
  The C++ compiler

    "/path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/build_stat_cxx_wrapper.sh"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/CMakeFiles/CMakeTmp

    Run Build Command(s):/path/to/bin/gmake cmTC_71648/fast && /path/to/bin/gmake  -f CMakeFiles/cmTC_71648.dir/build.make CMakeFiles/cmTC_71648.dir/build
    gmake[1]: Entering directory '/path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_71648.dir/testCXXCompiler.cxx.o
    /path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/build_stat_cxx_wrapper.sh -x c++   -ccbin xlc++ -qxflag=disable__cplusplusOverride   -std=gnu++14 -o CMakeFiles/cmTC_71648.dir/testCXXCompiler.cxx\
.o -c /path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    /path/to/tpls/xl/xl-2020.03.18/xlC/16.1.1/bin/.orig/xlC_r: warning: 1501-216 command option -bin is not recognized - passed to ld
    /path/to/tpls/xl/xl-2020.03.18/xlC/16.1.1/bin/.orig/xlC_r: warning: 1501-228 input file xlc++ not found
    gmake[1]: *** [CMakeFiles/cmTC_71648.dir/build.make:85: CMakeFiles/cmTC_71648.dir/testCXXCompiler.cxx.o] Error 252
    gmake[1]: *** Deleting file 'CMakeFiles/cmTC_71648.dir/testCXXCompiler.cxx.o'
    gmake[1]: Leaving directory '/path/to/jenkins/Trilinos-atdm-ats2-xl-2020.03.18_spmpi-rolling_serial_static_opt/SRC_AND_BUILD/BUILD/CMakeFiles/CMakeTmp'
    gmake: *** [Makefile:140: cmTC_71648/fast] Error 2

Current Status on CDash

Run the above query adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day.

Steps to Reproduce

One should be able to reproduce this failure as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

and the system-specific instructions at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

Just log into any of the associated machines and copy and paste the full CDash build name <build-name> listed above and run commands like:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh <build-name>

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<package-name>=ON \
 $TRILINOS_DIR

$ make NP=16

$ <command-to-run-on-compute-node> ctest -j4

where <package-name> is any package that you want to enable to reproduce build and/or test results.

Again, for exact system-specific details on what commands to run to build and run tests, see:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

If you can't figure out what commands to run to reproduce the problem given this documentation, then please post a comment here and we will give you the exact minimal commands.

[bartlettroscoe]

This is likely due to the upgrade to C++14 and/or the Kokkos upgrade. The XL builds were running just fine before that as shown here.

[e10harvey]

@jwillenbring: Who is the assignee for this?

[prwolfe]

@jwillenbring: Who is the assignee for this?

I am taking this as we discussed at stand-up

[prwolfe]

Looks like this failure is related to the commit 5262096. Essentially the added CXX_FLAGS set up a flag for the mvcc_wrapper that xlC does not understand. By experiment it looks like we need

-qxflag=disable__cplusplusOverride

for both builds and

-ccbin xlc++

only for the cuda build.

[prwolfe]

This is resolved. We are seeing other build failures which will be tracked separately.