[develop] Add GDAS nemsio data type for initial conditions

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

Added the capability for the SRW App to use GDAS nemsio data type as initial conditions. Ran a successful experiment with GDAS nemsio as the initial conditions on Jet: /mnt/lfs4/HFIP/hfv3gfs/Edward.Snyder/new-srw-dev-rufs/gdas-nemsio-add/expt_dirs/get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_nemsio_2020062900_ensemble_2mems.

While testing I noticed the retrieve_data.py script wasn't functioning properly. It would return a successful result when GDAS nemsio files weren't downloaded. Through some troubleshooting, I realized that the hpps_requested_files function returned an empty {} if the expected file list equaled the unavailable file list. Made minor change to the retrieve_data.py script to address this bug.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)
  get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_nemsio_2020062900_ensemble_2mems
  get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2mems
  get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600
  grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16

DEPENDENCIES:

DOCUMENTATION:

No document changes needed.

ISSUE:

This PR addresses issue #582

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

[mkavulich]

Good catch on the hpss retrieval bug, I had noticed problems with that too but hadn't had time to debug.

[MichaelLueken]

@EdwardSnyder-NOAA These changes look good to me! I was also able to successfully run the new get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_nemsio_2020062900_ensemble_2mems test on Jet. Approving these changes now.

[mkavulich]

Sorry for the whiplash, but I found a problem with this fix that causes failures elsewhere. Specifically the tests getting FV3GFS data for nemsio and netcdf are now failing. The ICS tasks complete successfully but the LBCS fail. It looks like the files are actually placed where they should be, but the script reports failure for some reason.

[mkavulich]

Had some offline conversation with @christinaholtNOAA, and it turns out this is actually the correct logic here. The arrays are not named very clearly.

[MichaelLueken]

@mkavulich - @EdwardSnyder-NOAA has reverted the changes to ush/retrieve_data.py as requested. I reran the get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_nemsio_2020062900_ensemble_2mems test and it still passes following this modification. I'll rerun the Jenkins tests in the morning (hopefully Gaea will be back). Before I rerun the Jenkins tests, however, I'd like to make sure that you approve of the latest modification. Thanks!

[mkavulich]

Thanks @MichaelLueken and @EdwardSnyder-NOAA

[MichaelLueken]

@mkavulich @EdwardSnyder-NOAA Following the latest update:

Two tests are now failing on Jet:

grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2 is segfaulting in make_ics_mem000 and grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR is segfaulting in make_lbcs_mem000.

One test is failing on Hera GNU:

The MET_verification_only_vx test is failing with the following error message:

ERROR: run_metplus failed: [Errno 17] File exists: '/scratch1/NCEPDEV/stmp2/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-718/expt_dirs/MET_verification_only_vx/2019061500/mem000/metprd/GridStat'

Manual reruns of the tests that failed all successfully passed. Once the Gaea tests pass, I will go ahead and move forward with merging this PR.

I know that in PR #732, the maxtries for task_run_MET_GridStat_vx_APCP#ACCUM_HR#h_mem#mem# was increased to 2. Should we also add maxtries: '2' to the rest of the verification tasks, as well as to parm/wflow/coldstart.yaml for the make_ics/lbcs tasks as well, to mitigate potential segfaults in the future?

[mkavulich]

@MichaelLueken I am hopeful that #683 will fix these intermittent failure problems, but if not I can include a bump to maxtries for all verification tasks and we can address the root problem in the future.

[MichaelLueken]

The tests on Gaea successfully passed, so I will now move forward with merging this work.