The SRW App's grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR WE2E test fails following the changes introduced in hash 889254a

[MichaelLueken]

Description

Following the update in hash 889254a, associated with PR #1658, the grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR WE2E test for the SRW App no longer runs. The test fails with the following message:

 Range_check Warning:DELP max =    186149.5      min =    44.24700
 Range violation on: 2019/ 7/ 1  1:28:40
Warn_K=  36 (i,j)= 1371  527 (lon,lat)=235.094  58.114 DELP =**********
     K=  35   7637.79053
     K=  37   **********
NOTE from PE     0: ==> Error from range_check: data out of bound
 Range_check Warning:TA max =    369.7302      min =    199.1034
 Range violation on: 2019/ 7/ 1  1:28:40
Warn_K=  36 (i,j)= 1370  527 (lon,lat)=235.051  58.129 TA = 369.73016
     K=  35    199.10344
     K=  37    257.54221
NOTE from PE     0: ==> Error from range_check: data out of bound

The out of bound message from the range_check suggests that the issue is a CFL violation. However, the RRFS_AK_3km grid's DT_ATMOS is currently set to 10. Lowering this value doesn't make sense (and values from 3 to 9 were tested and still failed with the same message). Ultimately, @mkavulich found that using larger values for DT_ATMOS allows the test to successfully run. Using a DT_ATMOS value of 30 was decided to move forward. This is weird and doesn't make sense, if this is truly an issue of CFL violation.

To Reproduce:

This issue can be found on Jet and Hera (and probably every other machine as well).

1. Clone the SRW App - git clone git@github.com:ufs-community/ufs-srweather-app.git
2. Update Externals.cfg to use hash 889254a for [ufs-weather-model]
3. ./manage_externals/checkout_externals
4. ./devbuild.sh -p=hera
5. module use $PWD/modulefiles
6. module load wflow_hera
7. conda activate regional_workflow
8. cd tests/WE2E
9. ./run_WE2E_tests.py -t=grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR -m=hera -a=epic

Additional context

A correction to the problem in the CCPP is needed to move forward with updating the ufs-weather-model hash in the SRW App. Please see PR #799 in the SRW App for details on this issue.

Output

 Range_check Warning:DELP max =    186149.5      min =    44.24700
 Range violation on: 2019/ 7/ 1  1:28:40
Warn_K=  36 (i,j)= 1371  527 (lon,lat)=235.094  58.114 DELP =**********
     K=  35   7637.79053
     K=  37   **********
NOTE from PE     0: ==> Error from range_check: data out of bound
 Range_check Warning:TA max =    369.7302      min =    199.1034
 Range violation on: 2019/ 7/ 1  1:28:40
Warn_K=  36 (i,j)= 1370  527 (lon,lat)=235.051  58.129 TA = 369.73016
     K=  35    199.10344
     K=  37    257.54221
NOTE from PE     0: ==> Error from range_check: data out of bound

Tagging @joeolson42 to see if this behavior is expected following the changes introduced in PR #43 of the ccpp-physics repository.

[MichaelLueken]

When PR #1754 has merged, this issue was also corrected. Closing issue now.