Use Python tool for get_extrnl_mdl_file tasks #681

christinaholtNOAA · 2022-02-22T18:40:17Z

DESCRIPTION OF CHANGES:

These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file.

The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools.

I "reviewed" the PR to leave comments where it might be helpful for reviewing.

TESTS CONDUCTED:

I ran all the 25km tests in grid_* and nco_* on Hera and was successfully able to stage data from disk as previously expected. Several jobs failed in the same way as had been previously documented.

On Jet, I tested the same set of 25 km tests using the sparse data staged on disk to test the alternate data_locations available on that platform (known data streams for Jet, HPSS, BDP, and NOMADS). These tests completed as expected -- some ran to completion by finding the data they needed on HPSS or AWS, while others failed to find any data and returned a correct non-zero exit status.

I pushed a couple of additional changes to retrieve_data.py as the "alternate data locations" logic was tested and needed minor modification.

Have CLI accept a list of file name templates. Have the script write out a summary file necessry for downstream tasks.

Fix summary file output Fix functionality for using disk without known location.

jobs/JREGIONAL_GET_EXTRN_MDL_FILES

modulefiles/tasks/cheyenne/get_extrn_ics

christinaholtNOAA · 2022-02-22T18:51:11Z

scripts/exregional_get_extrn_mdl_files.sh


-for fp in "${extrn_mdl_fps_on_disk[@]}"; do


I did not preserve the age checking functionality this script enables. If required, I can add it to the Python tool, but I am assuming that partial files do not show up in the real time data streams, and we shouldn't have to worry about that for staged data.

When the real-time data files are either created on Jet, or transferred from WCOSS, I'm pretty sure they populate chunk by chunk, so there's still the chance of the script grabbing a file that is incomplete. For staged data, I agree, that shouldn't be an issue. Is it hard to add this functionality back in for real-time data?

@christinaholtNOAA, pinging you on this question. Thanks!

I can work on adding it back if it's necessary.

I think it's important to have for users who will be sourcing real-time data that are copied over from WCOSS to other NOAA HPC.

@JeffBeck-NOAA Do we want this script to handle that, or Rocoto via dependency? Does it make sense to start a task before the data is ready?

@christinaholtNOAA, good question. Since the Python script is called through Rocoto, maybe a time dependency for the data in question could be added to the task in the XML? That would ensure the task isn't kicked off before the data is ready. What do you think?

@JeffBeck-NOAA - You should never use a time dependency to signal data availability. That is an anti-pattern that is guaranteed to fail at some point. Always use a data dependency on the files you are looking for with an appropriate "age" attribute to account for files that get populated over a period of time.

@christinaholtNOAA - I don't think it is ever safe to make assumptions about how data is populated via various delivery services. If you're wrong, tasks will get triggered before data is truly available and may fail depending on the race between how quickly the job starts to run and how quickly the data is populated. I think it's better to have some assurance the data really is available. Checking how long ago the file was last modified is one way to do it. There may be other approaches that are as good or better.

@christopherwharrop-noaa Do you prefer a data dependency or a check in this script? I was leaning toward a datadependency since there is currently no dependency on the get_extrn_* tasks now. The current behavior of the workflow in no way supports near-real time runs (retro-only is hard coded in XML template, and there are no dependencies on getting the data), so I was assuming that wouldn't be a feature I'd need to support with this refactor.

If we do intend to support real-time runs, many new features will be needed for that. I'm happy to add a datadep as part of this work, and to add a second checker inside the Python tool if we feel those are necessary now. I tend to go for "build it when it's needed", though.

christinaholtNOAA · 2022-02-22T18:59:19Z

scripts/exregional_make_lbcs.sh

@@ -124,7 +124,7 @@ fi
 #-----------------------------------------------------------------------
 #
 extrn_mdl_staging_dir="${CYCLE_DIR}/${EXTRN_MDL_NAME_LBCS}/for_LBCS"
-extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${EXTRN_MDL_LBCS_VAR_DEFNS_FN}"
+extrn_mdl_var_defns_fp="${extrn_mdl_staging_dir}/${EXTRN_MDL_VAR_DEFNS_FN}"


Changing several of these variables to the generic version (without ICS or LBCS) since they are stored in locations that are specific to ICS or LBCS.

christinaholtNOAA · 2022-02-22T19:00:36Z

tests/WE2E/run_WE2E_tests.sh

-      EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}"
+      EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
+    else
+      EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"


The retrieve_data.py script assumes nothing about the directory structure of the staged data, so the entire path must be templated here. The same is true below for the LBCS.

christinaholtNOAA · 2022-02-22T19:14:09Z

ush/retrieve_data.py

-            loc = os.path.join(input_loc, file_name)
+        for file_template in file_templates:
+            loc = os.path.join(input_loc, file_template)
+            logging.debug(f'Full file path: {loc}')


I changed the behavior here slightly to make it easier to call from bash. The file_template is now just a list, whereas before it was meant to be the same YAML structure as in the data_locations.yml file. That was totally a pain, and unnecessary.

christinaholtNOAA · 2022-02-22T19:14:58Z

ush/retrieve_data.py

+    with open(summary_fp, "w") as summary:
+        summary.write(file_contents)
+
+


This is very specific to the SRW, and I'm not a huge fan of that, but am fine with it for now. Let me know if this is problematic.

christinaholtNOAA · 2022-02-22T19:15:39Z

ush/retrieve_data.py

+        msg = dedent(f'''No data stores have been defined for
+               {cla.external_model}!''')
+        if cla.input_file_path is None:
+            data_stores = ['disk']


This allows us to provide disk locations for data that may not have a defined known location.

christinaholtNOAA · 2022-02-22T19:16:25Z

ush/retrieve_data.py

@@ -672,9 +714,13 @@ def parse_args():
        '--input_file_path',
        help='A path to data stored on disk. The path may contain \
        Python templates. File names may be supplied using the \
-        --file_names flag, or the default naming convention will be \
+        --file_templates flag, or the default naming convention will be \


This was a more descriptive variable name, so I went ahead and made the shift as I hooked it into the scripts..

christinaholtNOAA · 2022-02-22T19:17:20Z

ush/templates/FV3LAM_wflow.xml

@@ -281,7 +281,6 @@ MODULES_RUN_TASK_FP script.
    <envar><name>PDY</name><value><cyclestr>@Y@m@d</cyclestr></value></envar>
    <envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>
    <envar><name>CYCLE_DIR</name><value><cyclestr>&CYCLE_BASEDIR;/@Y@m@d@H</cyclestr></value></envar>
-    <envar><name>EXTRN_MDL_NAME</name><value>{{ extrn_mdl_name_ics }}</value></envar>


This information is unnecessary since it's included in the var_defns.sh file.

JeffBeck-NOAA · 2022-03-25T17:00:47Z

Thanks, @jwolff-ncar, those are the same paths as mine. It looks like we should use the wrfprs files for RAP, since that wrfnat path is no longer valid after/during 2019 at some point.

christinaholtNOAA · 2022-03-25T21:38:01Z

@JeffBeck-NOAA There are a couple of issues.

First, I am seeing files that the RAP files exist in that location under a different name. I can add that alternative template to look for, if you think that is necessary. The files on this date are named like this: gpfs_hps_nco_ops_com_rap_prod_rap.2019061412-17.wrf.tar

Second, the make_ics tasks fail. I am assuming because of the older version of HRRR.

If we are dropping support for anything other than operational RAP/HRRR, should we maybe be testing with a newer test case? I believe the failures that are showing up for this case are because of unsupported input instead of problems with the new data ingest method.

JeffBeck-NOAA · 2022-03-26T03:54:10Z

@christinaholtNOAA

First, I am seeing files that the RAP files exist in that location under a different name. I can add that alternative template to look for, if you think that is necessary. The files on this date are named like this: gpfs_hps_nco_ops_com_rap_prod_rap.2019061412-17.wrf.tar

Yeah, the RAP wrfnat file names apparently changed at some point between 2019 and 2021. The best option is to have the staging script check for both templates:

gpfs_hps_nco_ops_com_rap_prod_rap.YYYYMMDDHH-HH.wrf.tar

and

com_rap_prod_rap.YYYYMMDDHH-HH.wrf.tar

in the following location:

/NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD

Second, the make_ics tasks fail. I am assuming because of the older version of HRRR.

I was able to get the HRRR wrfprs data from the following location to run without an issue (for 2019061500 and 2021061500)

/NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_hrrr_prod_hrrr.YYYYMMDD_conusHH-HH.wrf.tar

Here are my two runs on Hera (I stopped them after chgres_cube completed):

/scratch2/BMC/fv3lam/beck/FV3-LAM/expt_dirs/HRRR_RAP_2019061500
/scratch2/BMC/fv3lam/beck/FV3-LAM/expt_dirs/HRRR_RAP_2021061500

If we are dropping support for anything other than operational RAP/HRRR, should we maybe be testing with a newer test case? I believe the failures that are showing up for this case are because of unsupported input instead of problems with the new data ingest method.

While a case from 2019 isn't too bad, it may be worth updating the WE2E test for HRRR grib2 files. The good news is the 2021 date above ran without a problem, so we know that the current HRRRv4 operational configuration works. I'm not sure what failure you ran into, but I agree, it's not from the new data ingest method.

christinaholtNOAA · 2022-03-28T15:43:18Z

I am running a test with the paths you provided, and will add those to the RAP/HRRR sections of the data_locations.yaml. And I have resolved the conflicts as of late last week.

If that works, what else still needs to be done? I'm not sure where this stands.

JeffBeck-NOAA · 2022-03-28T16:20:16Z

I am running a test with the paths you provided, and will add those to the RAP/HRRR sections of the data_locations.yaml. And I have resolved the conflicts as of late last week.

If that works, what else still needs to be done? I'm not sure where this stands.

I think that will be it! I'll run a final test, then approve, and we can get this merged.

I did want to ask about sourcing external model data from online locations. I saw that AWS will be supported, which is excellent! Were there plans to expand this to other data sources later? This could be leveraged in a future PR, although some sources may be redundant with what is available on AWS. I honestly am not sure we should invest more time in the previous NOMADS data staging feature, given it's propensity to be either down or extremely slow.

christinaholtNOAA · 2022-03-28T16:46:41Z

@JeffBeck-NOAA My make_ics step is still failing for this case, just as before with

 - FILE HAS            0  SOIL LEVELS
 - FATAL ERROR: COUNTING SOIL LEVELS.
 - IOSTAT IS:            0

Your case that ran successfully pulled data from disk, and it is unclear what input data it used. I reran the same configuration with 2021061600 (2 years into the future from the original case) and it also fails on make_ics in the same way as above.

Is it possible that this is related to some data magic that happens within the GSL DSG for the HRRR data stream?

JeffBeck-NOAA · 2022-03-28T16:58:03Z

@christinaholtNOAA, yes, I staged HRRR wrfprs files manually from this location on HPSS:

/NCEPPROD/hpssprod/runhistory/rhYYYY/YYYYMM/YYYYMMDD/com_hrrr_prod_hrrr.YYYYMMDD_conusHH-HH.wrf.tar

What files are you using for HRRR data and from where?

christinaholtNOAA · 2022-03-28T16:59:19Z

Okay, so you are using wrfprs files, not wrfnat files? That may be the difference. Let me give that a check.

JeffBeck-NOAA · 2022-03-28T16:59:44Z

Okay, so you are using wrfprs files, not wrfnat files? That may be the difference. Let me give that a check.

Yep, that'll do it. The HRRR wrfnat files don't contain soil information.

christinaholtNOAA · 2022-03-28T17:16:49Z

Okay, I've pushed all my changes. The last attempt at make_ics ran fine with the wrfprs files with HRRR ICS.

@JeffBeck-NOAA If you don't mind giving it another quick test, that'd be great.

JeffBeck-NOAA · 2022-03-28T21:21:22Z

Thanks, @christinaholtNOAA. I retested all external model combinations using FV3GFS, HRRR, and RAP, and they all staged correctly and worked with chgres_cube. Approving!

christinaholtNOAA · 2022-03-28T21:29:23Z

That's great news, @JeffBeck-NOAA! Thanks!

@gsketefian Are you okay with the changes I made in response to your review?

gsketefian · 2022-03-29T22:40:13Z

@christinaholtNOAA Thanks for your responses, I'm good with them. One more thing though. After looking through the PR comments, I did not see any mention of the tests that are for fetching files from HPSS. There are in regional_workflow/tests/WE2E/test_configs/wflow_features. They are:

get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019101818
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022518
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2020022600
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2021010100
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019101818
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022518
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2020022600
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021010100
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2021062000
get_from_HPSS_ics_GSMGFS_lbcs_GSMGFS
get_from_HPSS_ics_HRRR_lbcs_RAP
get_from_HPSS_ics_RAP_lbcs_RAP

The various dates with grib2 and nemsio are there because those are transition dates (e.g. where the directory naming convention changes or something similar). Do you mind running these as well just to be sure all is working (unless you ran them already and I missed it)? They are all on the 25km CONUS grid and should go pretty quick. Thanks.

christinaholtNOAA · 2022-03-29T22:46:52Z

@gsketefian Sure. I can run them.

christinaholtNOAA · 2022-04-08T14:45:31Z

@gsketefian I ran the tests you requested. They required some changes to accommodate the situation where "atmos" is included in the path. I've pushed those changes and passed all those hpss tests on Hera.

I am running the other 25km tests again to make sure these changes don't break anything there. I'd like to merge this once I can confirm those pass.

…' into data_ingest_tasks

christinaholtNOAA · 2022-04-12T18:19:34Z

I can now confirm that all the 25 km tests we've discussed above pass on Hera.

gsketefian · 2022-04-12T19:18:08Z

@christinaholtNOAA Great! Let's merge it.

@mkavulich

* Add missing user-defined stochastic physics options; fix stochastic physics seed generation script (ufs-community#704) ## DESCRIPTION OF CHANGES: Add missing user-defined options for tendency-based stochastic physics and fix the ensemble-based seed generation script to work regardless of whether stochastic physics is turned on or not. ## TESTS CONDUCTED: Tested on Hera using the following WE2E configurations with and without stochastic physics: config.grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh config.community_ensemble_2mems.sh ## ISSUE (optional): [Issue ufs-community#702](ufs-community#702) ## CONTRIBUTORS (optional): Thanks to @mkavulich and @chan-hoo for finding this problem. * Add namelist option for netCDF4 when running with the 3-km NA domain; update NAM HPSS settings and WE2E tests (ufs-community#707) * Change to netcdf4 when using the NA 3-km domain * Update HPSS paths for NAM data * Update NAM HPSS locations and dates for WE2E tests. * Remove lines from merge. * Tweaks to allow compiler and build_env_fn to be specified in the run_WE2E_test.sh script (ufs-community#711) * Changed 20200304 to 20200303 in ush/mrms_pull_topofhour.py (ufs-community#712) * Remove unused rocoto directory in ush (ufs-community#720) * Fix bug for nco we2e tests on Orion; re-organize we2e input data and nco we2e tests (ufs-community#713) * Update machine script for orion * Update machine script for wcoss_dell_p3 * Update we2e run script for wcoss and orion * Reorganize nco we2e tests * remove machine based logic * Add symlink for nco inline post test * Added stand-alone verification scripts (feature/issue_683_standaloneVX) (ufs-community#726) * Grid-stat and point-stat run scripts. * Stand-alone scripts for verification. * Added comments to gridvx scripts. * Added qsub_job.sh and added comments to provide context on running Vx. * remove machine base logic (ufs-community#727) * Allow user-defined file names for input template files (ufs-community#717) * Allow multiple template names * parameterize file_TMPL_FN and add a we2e test * Increase maxtries_task for make_grid/orog/sfc_climo * Modify file name and description * Changes to RRFS 3- and 13-km domains, setup.sh script bug fixes, make_ics task modification, and tweaks to stochastic physics namelist settings (ufs-community#721) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Change sotyp_from_climo to "true" based on operational RAP grib2 files. * Update for changes to stochastic physics namelist options. * Check for DO_ENSEMBLE="TRUE" when running ensemble verification and turn of VX when running in NCO mode. * Revert to 3-km domain. * Remove commented-out GFDL grid for the RRFS_NA_13km domain * Add RRFS_NA_13km WE2E test * Changes to comments. * Adding 25 km tests to Jet/Hera suites. (ufs-community#718) * Add a small 3km predefined grid over Indianapolis for testing (ufs-community#725) * Add 3km grid over Indianapolis. This is about 600km x 600km in extent (200 x 200 grid points). It is intended for use in the WE2E tests. * Edit comments. * Use Python tool for get_extrnl_mdl_file tasks (ufs-community#681) These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file. The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools. * Increase size of RRFS CONUS grid (ufs-community#724) Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov> * add include-style quality mark options in metplus confs (ufs-community#738) * Add Gaea as a supported platform for the regional_workflow (ufs-community#734) * Updates to port regional workflow to gaea * Temp change with -v as batch option * new fixes for gaea/slurm * Updated time for make lbcs * added TEST data directory path * Update gaea.sh * fixes for PR * Add more parameters to CSV file containing WE2E test info (ufs-community#740) ## DESCRIPTION OF CHANGES: The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests. Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description. In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file. This additional information consists of the values of the following experiment variables for each test: ``` PREDEF_GRID_NAME CCPP_PHYS_SUITE EXTRN_MDL_NAME_ICS EXTRN_MDL_NAME_LBCS DATE_FIRST_CYCL DATE_LAST_CYCL CYCL_HRS INCR_CYCL_FREQ FCST_LEN_HRS LBC_SPEC_INTVL_HRS NUM_ENS_MEMBERS ``` In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members). ## TESTS CONDUCTED: The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`. This created a new CSV file that contained the new fields (columns). The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct. ## DOCUMENTATION: The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`. This PR slightly modifies that documentation to update it. * Update directory structure of NCO mode (ufs-community#743) * update vertical structure of NCO mode * update sample script for nco * Fix typo on write component of new RRFS CONUS * Default CCPP physics option is FV3_GFS_v16 (ufs-community#746) * Updated the default CCPP physics option to FV3_GFS_v16 * Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local> * Adds an alternative python workflow generation path (ufs-community#698) * Workflow in python starting to work. * Use new python_utils package structure. * Some bug fixes. * Use uppercase TRUE/FALSE in var_dfns * Use config.sh by default. * Minor bug fixes. * Remove config.yaml * Update to the latest develop * Remove quotes from numbers in predef grid. * Minor bug fix. * Move validity checker to the bottom of setup * Add more unit tests. * Update with python_utils changes. * Update to latest develop additions (Need to re-run regression test) * Use set_namelist and fill_jinja_template as python functions. * Replace sed regex searches with python re. * Use python realpath. * Construct settings as dictionary before passing to fill_jinja and set_namelist * Use yaml for setting predefined grid parameters. * Use xml parser for ccpp phys suite definition file. * Remove more run_command calls. * Simplify some func argument processing. * Move different config format parsers to same file. * Use os.path.join for the sake of macosx * Remove remaining func argument processing via os.environ. * Minor bug fix in set_extrn_mdl_params.sh * Add suite defn in test_data. * Minor fixes on unittest on jet. * Simplify boolean condition checks. * Include old in renaming of old directories * Fix conflicting yaml !join tag for paths and strings. * Bug fix with setting sfcperst dict. * Imitate "readlink -m" with os.path.realpath instead of os.readlink * Don't use /tmp as that is shared by multiple users. * Bug fix with cron line, maintain quotes around TRUE/FALSE. * Update to latest develop (untested) * Bug fix with existing cron line and quotes. * Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR. * Update to latest develop * More updates. * Bug fix thanks to @willmayfield! Check both starting/ending characters are brackets for shell variable to be considered an array. * Make empty EXPT_BASEDIR workable. * Update to latest develop * Update in predef grid. * Check f90nml as well. Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu> * Fix typo and crontab issue on wcoss dell in workflow python scripts (ufs-community#750) * Fix typo and failure on wcoss * fix new line issue on wcoss dell * remove capture_output * Get USER from environment Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov> * Add new WE2E configs (ufs-community#748) ## DESCRIPTION OF CHANGES: Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. ## TESTS CONDUCTED: Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance. ## DEPENDENCIES: None. ## DOCUMENTATION: No documentation changes are required. * Added a fixed WoF grid and the python tool to determine the write component parameters (ufs-community#733) * Added a fixed WoF grid and the python tool to determine the write component parameters * Update set_predef_grid_params.sh * Renamed file as recommended and removed unused lines * Modified comment Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com> Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl> * Replace env with modulefiles in scripts (ufs-community#752) * change env to mod * update we2e script * WE2E script improvements for usability (ufs-community#745) ## DESCRIPTION OF CHANGES: * Modifications to `run_WE2E_tests.sh`: * Add examples to help/usage statement * Modifications to `check_expts_status.sh`: * Add arguments list that can be processed by `process_args` * Add new optional arguments: `num_log_lines`, `verbose` * Include a help/usage message ## TESTS CONDUCTED: * Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message. * Ran `check_expts_status.sh --help` from the command line and got the expected help message. * Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected. * Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message. ## DEPENDENCIES: PR #[241](ufs-community/ufs-srweather-app#241) ## DOCUMENTATION: A lot of this PR is documentation in the scripts. There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app. * Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (ufs-community#744) * Bug fixes (grid size + suppress screen output from module load) (ufs-community#756) ## DESCRIPTION OF CHANGES: 1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ). 2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context). ## TESTS CONDUCTED: 1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does). 2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is). * Update default SPP ISEED array in config_defaults.sh to use unique values (ufs-community#759) * Modify RRFS North America 3- and 13-km domain configuration and WE2E test. * Modify default ISEED values for SPP * Fix grid in WE2E test * Update workflow python scripts (ufs-community#760) * update python scripts * Change output file name of run_post to meet NCO standards (ufs-community#758) * change output file name * change variable name * update python script * remove duplicates * add a check for empty variables * move variable to common area * clean up unnecessary comments * update scripts * remove duplicate * update python scripts * fix user-staged dir path issue in python script * Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (ufs-community#763) * Add new var to we2e tests for new grids * rename we2e tests for custom grid * remove unnecessary $ * Modifications to `CODEOWNERS` file (ufs-community#757) * Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them. * Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo. * Adding a python utility for summarizing compute. (ufs-community#769) Adds a utility that summarizes Rocoto database computational usage information. * Add github actions for python unittests. (ufs-community#747) * Add github actions for python unittests. * Include all python script in ush * Skip defining QUILTING params when it is set to False * Update py_workflow * Update unittest for set_extrn_mdl_params. * Updates from develop. Co-authored-by: Daniel Shawul <dshawul@yahoo.com> * Update sample script for NCO mode (ufs-community#771) * update config.nco.sh * Add comment * Feature/noaacloud (ufs-community#767) * updates for noaacloud * working version * fixes for noaacloud * added extra modules for post * removed cheyenne-specific crontab editing section (ufs-community#773) * Pin down hera miniconda3 module file version. (ufs-community#770) Pin down the version of miniconda3 on Hera, and do not append to the module path. * update staged data dir (ufs-community#774) Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com> Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com> Co-authored-by: michelleharrold <harrold@ucar.edu> Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com> Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com> Co-authored-by: BenjaminBlake-NOAA <52074832+BenjaminBlake-NOAA@users.noreply.github.com> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov> Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov> Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com> Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local> Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com> Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu> Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov> Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Co-authored-by: Yunheng Wang <47898913+ywangwof@users.noreply.github.com> Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl> Co-authored-by: Michael Kavulich <kavulich@ucar.edu> Co-authored-by: Daniel Shawul <dshawul@yahoo.com>

christinaholtNOAA added 16 commits February 2, 2022 02:14

WIP

4715f05

Remove unused file.

7a35299

Updating jet module files.

76ee572

Updating hera module files.

c63d92b

Updating cheyenne module files.

101d39a

Updating orion module files.

187aca9

Call retrieve_data.py at run time.

e79bc21

Update test script to define input location.

c908fb2

Updating the defaults.

3a91165

Update retrieve_data.py

7faa5e7

Have CLI accept a list of file name templates. Have the script write out a summary file necessry for downstream tasks.

Updating common tasks for each machine.

20aca9a

Adding support for GSMGFS

ed57e56

retrieve_data updates.

f37e6d7

Fix summary file output Fix functionality for using disk without known location.

Separate logic for additional args.

99387be

Configure for correct disk locations.

b92b7e4

Merge remote-tracking branch 'origin/develop' into data_ingest_tasks

5688a7e

christinaholtNOAA requested review from mkavulich, gsketefian, JeffBeck-NOAA, RatkoVasic-NOAA and BenjaminBlake-NOAA as code owners February 22, 2022 18:40

christinaholtNOAA commented Feb 22, 2022

View reviewed changes

Clean up unused code. Reduce duplication.

95ab89c

christinaholtNOAA requested review from jwolff-ncar, chan-hoo, christopherwharrop-noaa, danielabdi-noaa, dmwright526, jkbk2004 and panll March 4, 2022 16:10

Include additional ops data paths. Use wrfprs.

ce63b19

JeffBeck-NOAA approved these changes Mar 28, 2022

View reviewed changes

JeffBeck-NOAA mentioned this pull request Mar 31, 2022

Changes to RRFS 3- and 13-km domains, setup.sh script bug fixes, make_ics task modification, and tweaks to stochastic physics namelist settings #721

Merged

christinaholtNOAA added 2 commits April 8, 2022 14:40

Try multiple archive internal directories.

8a4d4b6

Add netcdf files to FV3GFS.

31f6050

christinaholtNOAA added 3 commits April 8, 2022 14:53

Merge remote-tracking branch 'ufs-community_regional_workflow/develop…

e06eec9

…' into data_ingest_tasks

Fix botched merge.

29f977f

Fix another merge botch.

2cf85d5

JeffBeck-NOAA approved these changes Apr 12, 2022

View reviewed changes

gsketefian approved these changes Apr 12, 2022

View reviewed changes

christinaholtNOAA merged commit 79645c5 into ufs-community:develop Apr 12, 2022

christinaholtNOAA mentioned this pull request Apr 19, 2022

Tests run, but don't pull from disk as expected. #741

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Python tool for get_extrnl_mdl_file tasks #681

Use Python tool for get_extrnl_mdl_file tasks #681

christinaholtNOAA commented Feb 22, 2022 •

edited

Loading

christinaholtNOAA Feb 22, 2022

JeffBeck-NOAA Feb 28, 2022 •

edited

Loading

JeffBeck-NOAA Mar 8, 2022

christinaholtNOAA Mar 14, 2022

JeffBeck-NOAA Mar 15, 2022

christinaholtNOAA Mar 16, 2022

JeffBeck-NOAA Mar 17, 2022

christopherwharrop-noaa Mar 17, 2022

christopherwharrop-noaa Mar 17, 2022

christinaholtNOAA Mar 21, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

christinaholtNOAA Feb 22, 2022

JeffBeck-NOAA commented Mar 25, 2022

christinaholtNOAA commented Mar 25, 2022

JeffBeck-NOAA commented Mar 26, 2022 •

edited

Loading

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022 •

edited

Loading

christinaholtNOAA commented Mar 28, 2022

gsketefian commented Mar 29, 2022

christinaholtNOAA commented Mar 29, 2022

christinaholtNOAA commented Apr 8, 2022

christinaholtNOAA commented Apr 12, 2022

gsketefian commented Apr 12, 2022

		with open(summary_fp, "w") as summary:
		summary.write(file_contents)

Use Python tool for get_extrnl_mdl_file tasks #681

Use Python tool for get_extrnl_mdl_file tasks #681

Conversation

christinaholtNOAA commented Feb 22, 2022 • edited Loading

DESCRIPTION OF CHANGES:

TESTS CONDUCTED:

Choose a reason for hiding this comment

JeffBeck-NOAA Feb 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeffBeck-NOAA commented Mar 25, 2022

christinaholtNOAA commented Mar 25, 2022

JeffBeck-NOAA commented Mar 26, 2022 • edited Loading

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022

christinaholtNOAA commented Mar 28, 2022

JeffBeck-NOAA commented Mar 28, 2022 • edited Loading

christinaholtNOAA commented Mar 28, 2022

gsketefian commented Mar 29, 2022

christinaholtNOAA commented Mar 29, 2022

christinaholtNOAA commented Apr 8, 2022

christinaholtNOAA commented Apr 12, 2022

gsketefian commented Apr 12, 2022

christinaholtNOAA commented Feb 22, 2022 •

edited

Loading

JeffBeck-NOAA Feb 28, 2022 •

edited

Loading

JeffBeck-NOAA commented Mar 26, 2022 •

edited

Loading

JeffBeck-NOAA commented Mar 28, 2022 •

edited

Loading