Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flip atmos/memXXX to memXXX/atmos #1201

Merged

Conversation

aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Dec 23, 2022

Description

This PR:

  • flips the ensemble member directories which are subdirectories under components to having all the components for any member under the member directory.
  • Fixes Correct the hierarchy of ensemble members in ROTDIR #1196
  • The "atmos" jobs define "COMPONENT=atmos". They are "atmos" jobs, so there is no need to define a variable "COMPONENT" as it is obvious. Removes "${COMPONENT}" in the "atmos" jobs and make it explicit.

This PR also adds a "hack" to allow the success for the gdaspostanl job in the first half cycle.
The "hack" will be removed with the refactoring of the post jobs in the (near) future.

This change is necessary as GFSv17 will introduce coupled model components in the ensemble for initialization of the GEFS. Copying ICs for GEFS from GFS will be greatly simplified and the information for any given member will be self contained.

Since the changes in this PR are exhaustive, the linter recommendations will be addressed in a separate PR.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • A cycled experiment on Hera at C96/C48 ensemble has been conducted. Output can be found at:
EXPDIR: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/tmem01
ROTDIR: /work2/noaa/stmp/rmahajan/gwWork/ROTDIR/tmem01
  • A cycled experiment on Orion was also started and compared w/ develop. See this comment.
  • The hpssarch_gen.sh script is quite confusing and at this time, the archival of the ensemble data has been updated but has not been verified. Further PR's will address the issue. The earc step does succeed without errors. Since this was tested on Orion, it is unclear if this actually works on Orion in the first place. A cycled test on Hera/WCOSS2 might be necessary.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

scripts/exgdas_enkf_ecen.sh Outdated Show resolved Hide resolved
scripts/exglobal_atmos_analysis.sh Show resolved Hide resolved
scripts/exglobal_atmos_analysis.sh Show resolved Hide resolved
scripts/exgdas_enkf_earc.sh Show resolved Hide resolved
@aerorahul aerorahul marked this pull request as ready for review January 3, 2023 15:06
@aerorahul
Copy link
Contributor Author

Adding @XianwuXue-NOAA to ensure the early cycle EnKF does not run into issues with this branch.

Copy link
Contributor

@XianwuXue-NOAA XianwuXue-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested early-EnKF successfully after the change.

scripts/exgdas_enkf_sfc.sh Outdated Show resolved Hide resolved
@aerorahul
Copy link
Contributor Author

@CatherineThomas-NOAA @RussTreadon-NOAA
We would appreciate a review from either of you before we merge this PR.

@RussTreadon-NOAA
Copy link
Contributor

If the changes have been confirmed to not alter results in a cycled test of sufficient duration, the changes should be OK. If I were to review the changes this is what I would do. I would run at least two parallels. One would use g-w develop. The other would use aerorahul:feature/flip-memdir. Time and resources permitting I would also see if the changes in aerorahul:feature/flip-memdir reproduce the operational GFS.

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment #1201 (comment) for my feedback regarding these changes. Given confirmation that parallel test(s) have been run using aerorahul:feature/flip-memdir for sufficient duration with no change in cycled results, the changes in this PR should be merged into develop.

@aerorahul
Copy link
Contributor Author

@RussTreadon-NOAA
Thank you for your thoughtful comments.

Two experiments have been started on Orion:

control: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/devmem
experiment: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/expmem

The control uses develop and the experiment uses the branch from this PR.
The experiments are set to run for 4.5 cycles with GLDAS and MET OFF.
If this is not enough, then please let me know.

Running operational tests is not feasible as develop has significantly evolved from the operational tag.

I will do comparisons in the directories every cycle randomly for different members and report them here.

I will also attempt comparing HPSS tarballs. Though, that can be deferred to later PR as we are in the process of cataloging those files differently.

@RussTreadon-NOAA
Copy link
Contributor

The devmem and expmem parallels are still running on Orion. A spot check of files for 20211221/00 finds consistency between the two experiments. This is a good sign.

@aerorahul
Copy link
Contributor Author

aerorahul commented Jan 5, 2023

A few spot checks:

❯❯❯ date=2021122100; mem=013; file=gdas.t${date:8:2}z.atminc.nc; cmp devmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/atmos/mem$mem/$file expmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/mem$mem/atmos/$file
❯❯❯
❯❯❯ date=2021122106; mem=008; file=gdas.t${date:8:2}z.atmf006.nc; cmp devmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/atmos/mem$mem/$file expmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/mem$mem/atmos/$file
❯❯❯

Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orion parallels out to 2021122118. Spot check of 12Z cycle looks good. Give tentative approval. Will check more cycles later.

@RussTreadon-NOAA
Copy link
Contributor

Spot check of gfs, gdas, and enkfgdas at 20211222/00 shows consistency between the control devmem and the test expmem. Some files differ but these are expected and acceptable differences (e.g., master grib index files, loginc, loganl, gsi diagnostic files).

I discovered a new file, input.nml in gfs.$PDY/$cyc/atmos. It's in both devmem and expmem. forecast_postdet.sh copies this file to COMOUT when gfsfcst runs.

Copy link
Contributor

@CatherineThomas-NOAA CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for running the tests. Results look good to me as do the code changes themselves. Approve with the understanding that two aspects will be addressed in future PRs:

  • archival
  • top level non-member directory (enfkstat, spread, etc)

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 79e564a into NOAA-EMC:develop Jan 9, 2023
@aerorahul aerorahul deleted the feature/flip-memdir branch January 9, 2023 16:53
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 30, 2023
* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Correct the hierarchy of ensemble members in ROTDIR
5 participants