[Release/public-v2.2.0] Update data paths to v2p2 #937
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EdwardSnyder-NOAA
requested review from
mkavulich,
gsketefian,
JeffBeck-NOAA,
RatkoVasic-NOAA,
BenjaminBlake-NOAA,
ywangwof,
chan-hoo,
panll,
christinaholtNOAA,
christopherwharrop-noaa,
danielabdi-noaa,
mark-a-potts,
jkbk2004,
willmayfield,
dmwright526,
gspetro-NOAA,
natalie-perlin and
MichaelLueken
as code owners
| FIXlut: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/fix/fix_lut | ||
| FIXorg: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/fix/fix_orog | ||
| FIXsfc: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/fix/fix_sfc_climo | ||
| FIXshp: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/NaturalEarth | ||
| data: |
There was a problem hiding this comment.
Paths for ics_lbcs could be updateduse the pre-staged data on Cheyenne/Derecho in
/glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data (as in ./ush/machine/derecho.yaml)
ics_lbcs:
FV3GFS:
nemsio: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh}
grib2: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh}
netcdf: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/netcdf/${yyyymmdd}${hh}
NAM: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/NAM/${yyyymmdd}${hh}
HRRR: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/HRRR/${yyyymmdd}${hh}
RAP: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/RAP/${yyyymmdd}${hh}
GSMGFS: /glade/work/epicufsrt/contrib/UFS_SRW_data/v2p2/input_model_data/GSMGFS/${yyyymmdd}${hh}
```
| HRRR: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/develop/input_model_data/HRRR/${yyyymmdd}${hh} | ||
| nemsio: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh} | ||
| grib2: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh} | ||
| RAP: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/RAP/${yyyymmdd}${hh} |
There was a problem hiding this comment.
add netcdf: after line 38
netcdf: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/netcdf/${yyyymmdd}${hh}
| nemsio: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh} | ||
| grib2: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh} | ||
| RAP: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/RAP/${yyyymmdd}${hh} | ||
| HRRR: /lustre/f2/dev/role.epic/contrib/UFS_SRW_data/v2p2/input_model_data/HRRR/${yyyymmdd}${hh} |
There was a problem hiding this comment.
Add paths for GSMGFS:
ush/machine/hera.yaml
Outdated
There was a problem hiding this comment.
Add section with staged data before the rocoto
data:
ics_lbcs:
| nemsio: /contrib/EPIC/UFS_SRW_data/develop/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh} | ||
| grib2: /contrib/EPIC/UFS_SRW_data/develop/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh} | ||
| nemsio: /contrib/EPIC/UFS_SRW_data/v2p2/input_model_data/FV3GFS/nemsio/${yyyymmdd}${hh} | ||
| grib2: /contrib/EPIC/UFS_SRW_data/v2p2/input_model_data/FV3GFS/grib2/${yyyymmdd}${hh} |
There was a problem hiding this comment.
add other external model paths in
data:
ics_lbcs:
section (as in derecho.yaml )
| FIXlut: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/v2p2/fix/fix_lut | ||
| FIXorg: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/v2p2/fix/fix_orog | ||
| FIXsfc: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/v2p2/fix/fix_sfc_climo | ||
| FIXshp: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/v2p2/NaturalEarth |
There was a problem hiding this comment.
Add path for GSMGFS model in
data:
ics_lbcs:
MichaelLueken
added
the
run_we2e_coverage_tests
|
Submitting comprehensive Jenkins tests now. Please note that the comprehensive tests can only be launched manually in the Jenkins API. The |
|
The pipeline for the comprehensive tests for this PR can be found - |
|
@natalie-perlin I think I understand where your confusion is coming from. There are various old conventions for testing that are not explained anywhere, and because of this have not been followed by everyone. There also appear to be some outdated and unnecessary variables. The To clarify the need for I believe We should probably open an issue to discuss and resolve these problems. |
|
@EdwardSnyder-NOAA - please see issue There is a difference between the use of the variable When test or suite/tests from WE2E are run from ./tests/WE2E/ directory, their configuration includes using the variable TEST_EXTRN_MDL_SOURCE_BASEDIR to determine data locations for the get_ics and get_lbcs tasks, which is then stored as EXTRN_MDL_SOURCE_BASEDIR variable in config.yaml file For running a community test case on a Tier 1 system from ./ush directory using config.yaml script (copying from config.community.yaml), and launching generate_FV3LAM_wflow.py (copying from config.community.yaml), a variable TEST_EXTRN_MDL_SOURCE_BASEDIR is not used to set prestaged data locations for get_ics and get_lbcs tasks. The prestaged locations need to be in a "data: ics_lbcs" section of the .yaml file for the generated community case to search for data locally ("disk"), and not to attempt to pull it from hpss or aws. |
|
@EdwardSnyder-NOAA - The Hercules comprehensive tests failed (I suspect the Orion tests will fail as well). There are no orography fix files in: I'll let you know if or when other tests fail. |
|
@EdwardSnyder-NOAA - As noted above, the Orion comprehensive tests also failed since the necessary orography fix files are not available. |
|
@EdwardSnyder-NOAA - The comprehensive tests also failed on Hera (both Intel and GNU), because |
|
@EdwardSnyder-NOAA - There were failures in both Hera Intel and Hera GNU comprehensive tests.
The following tests failed:
How would you like to proceed with the testing now? I should note that the verification issues were also being encountered during manual runs of the comprehensive testing suite on Derecho. |
|
The following is a more detailed error message for the failed verification tasks: |
|
@EdwardSnyder-NOAA - I see that the verification observational data is a link from |
|
@natalie-perlin It seems like there is no need to set the |
|
Manually rerunning all tests that either failed or were never launched have passed, with the exception of Comprehensive testing on Hera GNU is now complete. |
|
Running the failed test, |
|
Some of the Hera GNU tests were failing because the |
|
The comprehensive tests have completed for Hera Intel: Once again, |
|
The comprehensive tests have completed on Jet: |
|
The Gaea C5 tests have completed: The only failures are related to pulling data from HPSS (which isn't available on Gaea C5) and |
|
The comprehensive tests have completed on Hercules: |
|
The comprehensive tests completed on Orion: A rerun of the However, I had to add Without the above modification, the get_extrn_* tasks will all fail with the following message in the log files: |
|
I have ran the comprehensive tests on the three noaacloud platforms using the newly added |
|
The comprehensive testing has completed on Gaea: The three verification tests were rerun and successfully passed after increasing the walltime for APCP tasks: |
|
The comprehensive tests on Derecho have completed: These results are identical to what @natalie-perlin found in her PR #894 (please this comment). In the above failed tests (not |
MichaelLueken
approved these changes
Oct 17, 2023
There was a problem hiding this comment.
@EdwardSnyder-NOAA - The comprehensive testing has been completed on all Tier-1 platforms via Jenkins and you have completed the testing on NOAA Cloud platforms. I have opened PR #944 to bring the changes not related to the machine file modifications into develop. This work is ready to be committed to the release/public-v2.2.0 branch and move the SRW v2.2 release into code freeze. Approving this PR now.
|
@natalie-perlin - I will go ahead and move forward with merging this PR into release/public-v2.2.0 at this time. As @mkavulich and @gspetro-NOAA have noted, the: sections are for archived data sets that might be staged on a given machine. These directories should not be the same as those found in the directories and the |
MichaelLueken
merged commit 86c4ea4
into
ufs-community:release/public-v2.2.0
3 checks passed
24 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
Update data paths to point the release staged data location for all supported platforms.
Type of change
TESTS CONDUCTED:
DEPENDENCIES:
DOCUMENTATION:
ISSUE:
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):