latest code runtime failures on cheyenne

[jedwards4b]

When using the ufs-mrweather-app hash de1ffe all tests are failing at runtime.
I tried replacing the output of chgres_cube, with that from an earlier version for which the model ran correctly. That also failed. Then I tried changing back to my local NCEPlibs install - because I built NCEPlibs with a newer version of the compiler I had to change that as well which then required me to also change the esmf build. With these changes the test I was trying now passes.

I'm am working to further pinpoint the cause. This discussion is continued from NOAA-EMC/fv3atm#43

[climbfuji]

I need to know please at which point the model fails and with which error message ... thank you.

[climbfuji]

I just completed the canned test case (aka simple-test-case, 24h fcst) on Cheyenne with the NCEPLIBS-external/NCEPLIBS installed as modules last week:

/glade/work/heinzell/fv3/ufs-weather-model/ufs-weather-model-ufs-public-release-20200210-for-amazon/simple-test-case

The modulesfiles for both GNU and Intel were missing the ESMFMKFILE entries, I added those.

This is how I compiled the latest release/public-v1 code using build.sh:

module purge
module load ncarenv/1.3
module load intel/18.0.5
module load ncarcompilers/0.5.0
module load mpt/2.19
module load netcdf/4.7.3
module use -a /glade/p/ral/jntp/GMTB/tools/modulefiles/intel-18.0.5/mpt-2.19/
module load NCEPlibs/1.0.0alpha01
export CMAKE_Platform=cheyenne.intel
./build.sh 2>&1 | tee build.log

This build uses the information in cmake/configure_cheyenne.intel.cmake and cmake/Intel.cmake, as well as cmake/FindESMF.cmake

[jedwards4b]

That was the problem, apparently the esmf_libs/8.0.0 esmf-8.0.0-ncdfio-mpi-g was incompatible with the NCEPlibs build - it's working after you fixed the ESMFMKFILE entry and I removed the other module.

[climbfuji]

Hooroay!