Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement fix reorg and remove gfs-utils code #1009

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Aug 31, 2022

Description
Removes all of the code and scripts that were moved to the new gfs-utils repo and adjusts workflow scripts to build and use them from the new location.

Some of the build scripts had unnecessary calls to machine-setup that are removed because the lower-level script already has the same functionality.

This PR also includes updates to use the new fix organization. This includes the addition of a fix versions file, updates to the link script, and some changes in the fix directories used by scripts to account for files that have been relocated. The versions file sets the version number for each component of fix, so that fix can be more easily maintained and documented. The initial versions are all the same, and correspond to the old fix_NEW directory (other than some directories have been renamed or reorganized). Except gdas, which has already had a new set of fix files added. The fix update also required an update to UFS-Utils.

Finally, the link script has been updated to match the syntax of the rest of the build system (checkout.sh and build_all.sh). link_workflow.sh now detects the machine automatically instead of requiring an argument, and dev mode, which used to be set using emc as an argument, is now assumed. To run in ops mode (copy instead of link), the -o option is used. The full syntax is now simply:

./link_workflow.sh [-o]

Fixes #356
Fixes #966
Depends on ufs-community/UFS_UTILS#688

**Type of change**
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [x] This change requires a documentation update

**How Has This Been Tested?**
- [x] 5½-cycle cycling test on Hera
- [x] Coupled test on Hera
- [x] 2½-cycle cycling test on Orion
- [x] Coupled test on Orion

**Checklist**
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] New and existing tests pass with my changes
- [x] Any dependent changes have been merged and published

@WalterKolczynski-NOAA WalterKolczynski-NOAA added feature New feature or request maintenance Regular updates and maintenance work labels Aug 31, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA self-assigned this Aug 31, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the blocked Issue is currently being blocked by another issue label Aug 31, 2022
@WalterKolczynski-NOAA
Copy link
Contributor Author

Currently using a temporary repo for UFS_UTILS until ufs-community/UFS_UTILS#688 is merged.

@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Feature/use gw utils Remove gfs-utils code and update for new fix Aug 31, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Remove gfs-utils code and update for new fix Update for new fix and remove gfs-utils code Aug 31, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Update for new fix and remove gfs-utils code Update for fix reorg and remove gfs-utils code Aug 31, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as ready for review September 20, 2022 20:07
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 72 potential problems in the proposed changes. Check the Files changed tab for more details.

@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as draft September 20, 2022 20:08
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 12 potential problems in the proposed changes. Check the Files changed tab for more details.

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the blocked Issue is currently being blocked by another issue label Sep 22, 2022
sorc/build_all.sh Fixed Show fixed Hide fixed
sorc/link_workflow.sh Fixed Show fixed Hide fixed
sorc/link_workflow.sh Fixed Show fixed Hide fixed
sorc/link_workflow.sh Fixed Show fixed Hide fixed
sorc/link_workflow.sh Fixed Show fixed Hide fixed
sorc/link_workflow.sh Fixed Show fixed Hide fixed
sorc/build_all.sh Fixed Show fixed Hide fixed
sorc/build_all.sh Fixed Show fixed Hide fixed
sorc/build_ww3prepost.sh Fixed Show fixed Hide fixed
@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as ready for review September 23, 2022 21:02
@WalterKolczynski-NOAA
Copy link
Contributor Author

Going to rerun a test or two over the weekend from a clean install to confirm things still work, but should be able to start reviewing.

@WalterKolczynski-NOAA
Copy link
Contributor Author

Re-tests pass

ush/gfs_bufr.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. Some comments.
This PR would have been much easier to review if it were separate PRs:

  1. removing gfs-utils
  2. fix reorg
  3. updates to build scripts
  4. updates to link_workflow.sh

sorc/link_workflow.sh Show resolved Hide resolved
sorc/link_workflow.sh Outdated Show resolved Hide resolved
sorc/link_workflow.sh Outdated Show resolved Hide resolved
sorc/link_workflow.sh Show resolved Hide resolved
sorc/link_workflow.sh Show resolved Hide resolved
sorc/link_workflow.sh Show resolved Hide resolved
ush/gfs_truncate_enkf.sh Show resolved Hide resolved
ush/fv3gfs_regrid_nemsio.sh Show resolved Hide resolved
ush/fv3gfs_remap.sh Show resolved Hide resolved
- Source new fix.ver file to get fix subfolder version values for setting up fix folder symlinks.
- Change "fix_NEW" to "fix" for FIX_DIR paths: use newly reorganized and versioning fix set.
- Remove "fix_" from /fix folder symlink names.
- Update /fix folder symlink creation to use version variables from fix.ver.
- Update GSI and GDAS fix symlinks.

Refs: NOAA-EMC#966
- Change FV3JEDI_FIX in related configs to change the path subfolder from
"fix_jedi" to "gdas".

Refs: NOAA-EMC#966
- Remove "fix_" text from fix file paths in several configs.

Refs: NOAA-EMC#966
KateFriedman-NOAA and others added 16 commits October 4, 2022 00:34
- Update /scripts and /ush scripts to remove the "fix_" text
from fix subfolder paths.

Refs: NOAA-EMC#966
- Create a new version folder to hold version files.
- Create a new fix.ver to hold fix file versions.

Refs: NOAA-EMC#966
Removes all of the code and scripts that were moved to the new gfs-utils
repo and adjusts workflow scripts to build and use them from the new
location.

Some of the build scripts had unnecessary calls to machine-setup that are
removed because the lower-level script already has the same functionality.

This PR also includes updates to use the new fix organization. This includes
the addition of a fix versions file, updates to the link script, and some
changes in the fix directories used by scripts to account for files that
have been relocated. The versions file sets the version number for each
component of fix, so that fix can be more easily maintained and documented.
The initial versions are all the same, and correspond to the old fix_NEW
directory (other than some directories have been renamed or reorganizated).

Finally, the link script has been updated to match the syntax of the rest
of the build system (`checkout.sh` and `build_all.sh`). `link_workflow.sh`
now detects the machine automatically instead of requiring an argument, and
dev mode, which used to be set using `emc` as an argument, is now assumed.
To run in ops mode (copy instead of link), the `-o` option is used. The
full syntax is now simply:
```
./link_workflow.sh [-o]

-o: Run in operations (NCO) mode (copy instead of link)
```

This commit uses a temporary clone of the UFS_UTILS repo pending the merger
of ufs-community/UFS_UTILS#688.

Fixes NOAA-EMC#356
An incorrect resolution of a merge conflict resulted in an undefined
variable in the link script.

Refs: NOAA-EMC#356
Updates the gfs utils version to pick up a fix for an incorrect gempak
module version on Orion.
An earlier change had inadvertently stopped checking out GSI Monitor,
GSI Utils, and GLDAS when GDAS is used instead of GSI.

Refs: NOAA-EMC#966
First wave of linter fixes for removing gfs-utils code/fix reorg.

Refs: NOAA-EMC#966
Second wave of linter fixes for removing gfs-utils code/fix reorg.

Refs: NOAA-EMC#966
Wave 3 of linter fixes for removing gfs-utils code/fix reorg.

Refs: NOAA-EMC#966
Wave 4 of linter fixes for removing gfs-utils code/fix reorg.

Refs: NOAA-EMC#966
The last merge from develop introduced a minor syntax error in the
link script due to a bad conflict resolution. The issue is now
corrected.

Refs: NOAA-EMC#966
GLDAS scripts were recent moved into the workflow repo and need to
be updated for the new fix structure like other components.

Refs: NOAA-EMC#622, NOAA-EMC#966
Corrects a `:-?` to `:?`.

Refs: NOAA-EMC#356
Moved the setting of FIXDIR higher so it errors on an unknown machine
sooner. Also removed redundant version file load.

Refs: NOAA-EMC#966
The closing quotation mark when making the logdir in checkout was
erroneously split into a new line.
@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Update for fix reorg and remove gfs-utils code Implement fix reorg and remove gfs-utils code Oct 4, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit e7f72e8 into NOAA-EMC:develop Oct 4, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA deleted the feature/use_gw_utils branch October 4, 2022 07:03
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 30, 2023
* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request maintenance Regular updates and maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix file promotion and updated versioned FIX_DIR paths in workflow
3 participants