Reorganize the WE2E (workflow end-to-end) testing system

[gsketefian]

DESCRIPTION OF CHANGES:

This PR reorganizes the WE2E testing system so that it is easier to comprehend, use, and modify. The major changes are as follows:

• Move the WE2E testing system from the regional_workflow/tests directory to regional_workflow/tests/WE2E. This is because we anticipate other types of test in the testing system, e.g. unit tests, which would go under regional_workflow/tests/unit, etc.
• Move the WE2E test configuration files that were in regional_workflow/tests/baseline_configs to category subdirectories under regional_workflow/tests/WE2E/test_configs. The category subdirectories thus far and the types of tests they contain are:
  • grids_extrn_mdls_suites_community
    Tests in community mode of various combinations of the grid, external models for ICs and LBCs, and physics suites.
  • grids_extrn_mdls_suites_nco
    Tests in NCO mode of various combinations of the grid, external models for ICs and LBCs, and physics suites.
  • release_SRW_v1
    The graduate student test (GST) used for the UFS SRW App version 1 release.
  • wflow_features
    Test of workflow features, e.g. ability to set various parameters to user-specified values instead of using the defaults, ability to fetch external model files from different models and on different dates from NOAA-HPSS, etc.
• Rename some of the WE2E test configuration files to adhere to the naming convention used in each category subdirectory.
• Remove some of the WE2E test configuration files since they are almost-duplicates, e.g. they differ with respect to another test only in the cycle date/time used or the LBC specification interval.
• Changes to contents of test configuration files:
  • Rearrange the order in which experiment variables are specified in the test configuration files so that the predefined grid name and the physics suite are set first, then the external model info is set, then the cycle dates, then the forecast length and LBC update interval, finally followed by other parameters.
  • Remove redundant variable specifications in the test configuration files, e.g. remove QUILTING="TRUE" since QUILTING is already set to "TRUE" by default, remove GRID_GEN_METHOD="ESGgrid" since a predefined grid is already specified that in turn implies a value for GRID_GEN_METHOD.
  • Include a test purpose/description in each test configuration file.
  • Perform bug fixes in the WE2E configuration files, e.g. a test that was supposed to run on the RRFS_CONUS_3km grid was actually running on the RRFS_CONUS_13km grid.
• Remove the file baselines_list.txt since it it not used.
• Add new function in get_WE2Etest_names_subdirs_descs.sh that:
  • Searches subdirectories under the base directory in which the WE2E test configuration files are located (regional_workflow/tests/WE2E/test_configs) and returns a list of all available test names, the category subdirectories under the base directory in which they are located, the unique test IDs, and the test descriptions.
  • Creates (if requested) a comma-separated value (CSV) file containing this WE2E information that can be opened as a spreadsheet in Google Sheets.
• Rename run_experiments.sh to run_WE2E_tests.sh.
• In run_WE2E_tests.sh:
  • Include a detailed usage message.
  • Make sure that required arguments are provided on the command line.
  • Call the new function get_WE2Etest_names_subdirs_descs to get a full list of all available tests. Then check the list of test names that the user wants to run to make sure all exist in the full list.
  • Run sanity checks on the user-specified list of tests to run, e.g. that a test is not repeated (either under the same name or under an alternate name).
• In ush/bash_utils/filesys_cmds_vrfy.sh, put the local attribute in front of variables that are supposed to be local.
• In ush/bash_utils/is_element_of.sh, add the :- at the end of array_name_at so that the function still works when the array passed in is empty.
• In ush/config_defaults.sh, edit comments and move groups of variables to more appropriate location in file.

TESTS CONDUCTED:

On Hera, ran the following tests so far:

• get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000
• inline_post
• specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
• specify_RESTART_INTERVAL

All were successful. Since it is expensive to run all the new WE2E tests, the remainder will be tested gradually and the results recorded in a google spreadsheet (for now; this will have to be automated at some point).

DEPENDENCIES:

There will likely be a companion PR into ufs-weather-app that updates the hash of regional_workflow, but that can only be created after this PR is merged.

DOCUMENTATION:

Detailed documentation for how the new WE2E testing system works is in the comments in the files run_WE2E_tests.sh and get_WE2Etest_names_subdirs_descs.sh. Transferring these to RST files will take time and is likely better to do as part of a separate PR.

[JeffBeck-NOAA]

Is quilting removed in these WE2E config files because "TRUE" is the default and/or because this option may be removed from the workflow in the near future?

[gsketefian]

Because QUILTING is set to "TRUE" in config_defaults.sh, so it doesn't need to be reset in the config files. No plans to remove QUILTING from the workflow!

[JeffBeck-NOAA]

Oh, OK, should it also be removed from config.community.sh?

[gsketefian]

I suppose yes, and config.nco.sh as well, but I didn't look at those since they're not part of the WE2E tests. I could, but I'd have to test them. I haven't tested most of these reorganized WE2E tests yet. I was hoping to get feedback as to which ones to test.