S2SW noaero test fails in Debug mode for GNU

[DeniseWorthen]

Description

The existing cpld_debug_noaero_p8 test is used to test S2S on GNU. Adding Waves to this test fails with an error in CCPP/physics:

  0: Using non-aerosol-aware version of Thompson microphysics
  0:    calling table_ccnAct routine
  0:    creating qc collision eff tables
  0:    creating rain evap table
  0:    creating ice converting to snow table
  0:    creating rain collecting graupel table
 24: At line 4310 of file /scratch2/NCEPDEV/climate/Denise.Worthen/WORK/ufs-weather-model-gnu/FV3/ccpp/physics/physics/module_mp_thompson.F90 (unit = 63, file = 'qr_acr_qgV2.dat')
 24: Fortran runtime error: End of file

Note this same error was found when compiling the branch containing the new WW3 cap, which can be run in debug mode. I think it most likely is an input file error.

To Reproduce:

On hera

1. check out the develop branch
2. in rt_gnu.conf, add Waves to the debug compile for the cpld_debug_noaero_p8 test

COMPILE | -DAPP=S2SW -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8

3. Make the following change in the WW3/CMakeLists.txt:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 1a7e7ab8..a76feea9 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,6 +17,7 @@ if(hasParent)
   # with Ww3
   set(CMAKE_Fortran_FLAGS "")
   set(CMAKE_C_FLAGS "")
+  remove_definitions(-DDEBUG)
 endif()

3. In the cpld_debug_noaero_p8 test, comment out the resource settings using cpl_c96. This will instead force the use of the cpl_dflt settings, which include the wave resources.

4. Turn waves on in the test

+#export CPLWAV=.false.
+#export CPLWAV2ATM=.false.
+#export MOM6_USE_WAVES=False
+export CPLWAV=.true.
+export CPLWAV2ATM=.true.
+export MOM6_USE_WAVES=True

5. Switch the nems.configure to one which includes waves (but not CHM)

+#export NEMS_CONFIGURE=nems.configure.cpld.IN
+export NEMS_CONFIGURE=nems.configure.cpld_wave.IN

6. Run the single test with GNU

A build directory is here: /scratch2/NCEPDEV/climate/Denise.Worthen/WORK/ufs-weather-model-gnu/
A run directory is here: /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/rt_753/cpld_debug_noaero_p8

[DeniseWorthen]

I can reproduce this error using the new wave cap and the gnu debug on hera. The error is identical as w/ the develop branch.

I copied my run directory from the gnu debug_noaero test and built the S2SW app in debug mode w/ intel and no aerosols.

./compile.sh hera.intel '-DAPP=S2SW -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8' '' YES NO 2>&1 | tee compile.log

I copied the executable and modules.fv3 and was able to run the noaero test with intel.debug. Run directory

/scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/rt_gnu_wave

[DeniseWorthen]

See Issue 1115 for the proposed resolution of this failure.

PR #1228 will resolve the endian-ness issue related to WW3 by allowing the endianness to be specified for WW3 only w/o impacting other components.

[junwang-noaa]

@DeniseWorthen Is this issue resolved?

[DeniseWorthen]

I will test this and let you know. If it works, I'll create a PR which replaces the S2SW with S2SWA debug test for GNU.

[DeniseWorthen]

I was able to run both cpld_control_p8 and cpld_debug_p8 on Cheyenne.gnu. I'll confirm those also work on hera.gnu and then create a PR.

[DeniseWorthen]

On Hera.GNU, cpld_control_p8 and cpld_debug_p8 tests now run to completion (but fail because of missing baseline). These two tests can be updated to "S2SWA".

I think we still need the S2S compilation and cpld_control_nowave_noaero_p8 test because it is used in CI.