Feature/met on hera #552
Feature/met on hera #552
Conversation
siwei-noaa
requested review from
BenjaminBlake-NOAA,
gsketefian,
JeffBeck-NOAA,
JulieSchramm and
mkavulich
as code owners
| # | ||
| METPLUS_PATH=\"${metplus_path}\" | ||
| MET_INSTALL_DIR=\"${met_install_dir}\"" | ||
|
|
There was a problem hiding this comment.
What's the point of these two lines?
METPLUS_PATH="${metplus_path}"
MET_INSTALL_DIR="${met_install_dir}""
I don't see where the lower-case versions of these variables are defined.
There was a problem hiding this comment.
I made a mistake in define the variable. line 804 and 805 should be the lower-case. I will correct this.
tests/run_experiments.sh
Outdated
| # | ||
| if [ "${RUN_TASK_VX_GRIDSTAT}" = "TRUE" ] || [ "${RUN_TASK_VX_POINTSTAT}" = "TRUE" ]; then | ||
| if [ "$MACHINE" = "HERA" ]; then | ||
| MET_INSTALL_DIR="/contrib/met/10.0.0" |
There was a problem hiding this comment.
I am not sure how the test worked if on lines 792 and 793, MET_INSTALL_DIR and METPLUS_PATH are defined (instead of met_install_dir and metplus_path), but on the right-hand sides of lines 804 and 805, the lower-case variables are used. That should have failed with an "undefined variable" error (because set -u is activated at the start of this script). Use the lowercase ones on lines 792-793.
There was a problem hiding this comment.
Also, the thing you should test is running this on a machine other than Hera to see if this error-catching actually works.
|
@siwei-noaa Can you also run this on a machine other than Hera to see if this error-catching actually works? Thanks. |
Yes, I can test this on Jet. But if we need to have observation data to obtain the wanted crash or not necessary? If you have ever run regional workflow on Jet? If so, I would like to mimic your configuration. Thank you. |
|
@siwei-noaa You don't need observation data because the workflow should not even be generated. You don't even need to build the code. Just clone it, do the manage_externals step, then try running run_experiments.sh just for the config.MET_verification.sh test. You should get an error (the error message you put in as part of this PR). We have run regional_workflow on Jet (@JeffBeck-NOAA runs it all the time), but you probably don't need to get all his configuration settings, etc. |
@gsketefian Could you please tell me how to run run_experiments.sh quickly? I just tried and got error. |
|
@siwei-noaa Can you point me to the directory on Jet in which you're running run_experiments.sh? What is the error you're getting (you're supposed to get an error)? |
|
The dir is: /mnt/lfs4/BMC/wrfruc/she/ufs-srweather-app/regional_workflow/ |
|
@siwei-noaa First, create under the
On Jet, I copied your directory to mine and ran the test, and it gave the expected error (see below), so you'r actually good to go! You can try the above command too to make sure you get the same behavior as I did. Approving now. |
|
Perfect, thank you. @gsketefian |
|
@siwei-noaa @gsketefian @JeffBeck-NOAA , Have you run this WE2E test on Hera? I got an error: model files does not exist: |
|
@chan-hoo I did run @siwei-noaa 's experiment on Hera (see my replay to him above). It should give the following error since MET and MET+ have not been properly set up on Hera: I did not get to the point of your error; it shouldn't get to that point. Do you have an experiment directory on Hera I can look at? |
|
@gsketefian, here is my dir on Hera: /scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/ufs_srw_app/srw_dev_test/expt_dirs/MET_verification |
|
@chan-hoo Sorry, I got confused. It's supposed to work on Hera and fail on other machines. I guess then this is a question for @siwei-noaa. @siwei-noaa, which date did you use for your test? |
|
@siwei-noaa , I think you might use your own config.sh file or didn't update the 'config.MET_verification.sh' file. In this file or 'run_experiments.sh', 'CCPA_OBS_DIR', 'MRMS_OBS_DIR', and 'NDAS_OBS_DIR' are not specified. I used the data in '/scratch2/BMC/det/UFS_SRW_app/v1p0/obs_data/' on Hera for testing the verification task. |
|
@chan-hoo I'm guessing @siwei-noaa didn't use the run_experiments.sh script but instead called generate_FV3LAM_wflow.sh using his own config.sh. But he can confirm. If so, this PR needs to be retested using run_experiments.sh. |
|
@gsketefian, I was adding wcoss part to run_experiments.sh. If you and @siwei-noaa agree, I'll update run_experiments.sh in my PR. |
|
@chan-hoo You mean in a new PR, you're going to add wcoss stanzas to run_experiments.sh? That works for me. But we still need to make sure that MET_verification works on Hera. It seems like a matter of just adding the FV3GFS files for 2019041500 to the staged-data directory. @siwei-noaa probably has those somewhere since he ran the case successfully somehow. And I guess also adding the variables 'CCPA_OBS_DIR', 'MRMS_OBS_DIR', and 'NDAS_OBS_DIR' somewhere, like @chan-hoo said. |
|
@gsketefian, ok. I'll wait until @siwei-noaa resolves this issue, and then update it for wcoss. |
|
@siwei-noaa @chan-hoo I created a new issue for this (#560). Things have changed a bit since I merged in my big WE2E reorg PR (#531). The issue has more info, so please take a look. Thx. |
|
@chan-hoo and @gsketefian, sorry for my late response as I had my annual leave last week. First, I would like to confirm some information about my WE2E test on Hera: (1) my test used date 20190415, (2) I did not run run_experiments.sh but call generate_FV3LAM_wflow.sh, (3) I used staged files from "/scratch2/BMC/fv3lam/RRFS_baseline/model_data/FV3GFS" and observed files from "/scratch2/BMC/fv3lam/RRFS_baseline/obs_data/". These directories only have files for our RRFS baseline experiments, but I do not know the directories for general use. Then, could you please tell me what should I do next? I saw https://github.com/NOAA-EMC/regional_workflow/issues/560 wants to include CCPA_OBS_DIR, MRMS_OBS_DIR, and NDAS_OBS_DIR. As I mentioned in the previous section, if we have a general directory that includes these observations? |
|
@siwei-noaa, as can be seen in 'regional_workflow/tests/WE2E/run_WE2E_tests.sh' (Lns 932-945), the user staged external data should be located in the designated directories. If you have permission, you can put the data for 20190415 into the directories. You should add the *_OBS_DIR parameters to somewhere between Lns 1019-1039. |
|
@siwei-noaa @chan-hoo I fetched the 20190415 nemsio tar file from HPSS, and it contained only the analysis, no forecast files. So LBCs couldn't be generated. That's why I then tried another data a year later -- 20200415. That worked, and the data is in that "staged_extrn_mdl_files" directory on Hera. Not sure how @siwei-noaa generated the LBCs. See Issue #560 for details. |
|
@gsketefian @chan-hoo I did not generate the LBCs. All these files are from Jamie Wolff (/scratch2/BMC/fv3lam/RRFS_baseline/model_data/FV3GFS), and they are ready to be used. |
|
@chan-hoo Thank you for your information on run_WE2E_tests.sh. May I know if /scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/staged_extrn_mdl_files is a permanent directory for staging file? I am asking because I think the dates of stage and observation files would vary case by case. Or should we stay on 20190415? |
|
@siwei-noaa On Hera, the directory I wonder how Jamie got the LBC files. She may have done something that is not doable as part of the workflow. @jwolff-ncar , can you say how you got the FV3GFS forecasts from which to generate LBCs? They are not in the nemsio tar file that the workflow gets from NOAA-HPSS. |
|
To keep the tests consistent and to avoid adding more data to disk, shouldn't we simply use the GST date (20190615)? The data we need for any WE2E test should be staged on hera at /scratch2/BMC/det/UFS_SRW_app/v1p0/obs_data/. The data we are using the the RRFS baselining was from Jili (we did not pull that data our selves). |
|
I agree with @jwolff-ncar. |
|
@jwolff-ncar Yes, that is fine. I just guessed another date thinking that's the time of year you wanted. The tests by default call the make_ics and make_lbcs tasks, so we would normally start from the FV3GFS files and generate the ICs and LBCs from those (instead of starting with pre-made IC and LBC files). @siwei-noaa, can you change the date in Thanks. Please let me know if you have any questions. |
|
@gsketefian Sure, I can do this. Should my changes based on https://github.com/NOAA-EMC/regional_workflow/issues/560 or the latest develop branch? |
|
@siwei-noaa Please start with the latest develop branch (any new PR you want to create you should start with the latest develop branch). |
|
@gsketefian Should I create 'my_tests.txt' myself? It seems this file does not exist. |
@siwei-noaa Yes, you have to create it (and call it whatever you want, just use the same name in the all to run_WE2E_tests.sh). It is just a text file containing the names of the tests you want to run. In this case, it contains just one line that says "MET_verification". |
|
@siwei-noaa, you can refer to my doc (Section 4.4): https://drive.google.com/file/d/12r9mSHSYgI5O3Whgeit9pIGmfIcoRm7T/view?usp=sharing and my sample file on my github: The structure of WE2E has been changed recently. So, it is not the same as that in the user's guide of SRW App. |
|
@gsketefian and @chan-hoo I see, thank you for the information. |
|
@gsketefian It seems "/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/staged_extrn_mdl_files/FV3GFS/" does not have the required IC files for the GST date (20190615). Should I still use GST date rather than 20200415? However, the issue of using 20200415 is that the observations are not available at "/scratch2/BMC/det/UFS_SRW_app/v1p0/obs_data/". |
|
The staged model data on hera should be here:
/scratch2/BMC/det/UFS_SRW_app/v1p0/model_data
We want to avoid using personal directories if at all possible.
…On Thu, Jul 29, 2021 at 9:48 AM Siwei He ***@***.***> wrote:
@gsketefian <https://github.com/gsketefian> It seems
"/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/staged_extrn_mdl_files/FV3GFS/"
does not have the required IC files for the GST date (20190615). Should I
still use GST date rather than 20200415? However, the issue of using
20200415 is that the observations are not available at
"/scratch2/BMC/det/UFS_SRW_app/v1p0/obs_data/".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/NOAA-EMC/regional_workflow/pull/552#issuecomment-889259076>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFN5ZDDM34UB3YGZUISIOHLT2FZ4NANCNFSM5AP5WM5Q>
.
--
Jamie K. Wolff
National Center for Atmospheric Research (NCAR)
Research Applications Laboratory (RAL)
Phone: 303.497.2812 Email: ***@***.***
*My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.*
|
|
Thank you for the information, @jwolff-ncar. However, it seems that this directory also misses certain required files, such as gfs.atmanl.nemsio. |
|
@siwei-noaa It looks like the files do exist in that directory that's in my space. They're here: I think the directory " Since my directory already contains the files you need, your test should work. Can you say what the error you're getting is? |
|
@gsketefian. I see. Here is the error: |
|
@siwei-noaa I think that error can be fixed if we specify the file format to be grib2. Can you replace the two lines in Thanks. |
DESCRIPTION OF CHANGES:
TESTS CONDUCTED:
A test run was conducted and has been finished successfully.
DEPENDENCIES:
To have MET verification run successfully, the observational data (e.g., CCPA, MRMS, NDAS) must be available.
DOCUMENTATION:
N/A
ISSUE (optional):
This is a follow up PR to complete the previous one in https://github.com/NOAA-EMC/regional_workflow/pull/537
CONTRIBUTORS (optional):
@gsketefian contributed the revision.