Convert archive jobs to proper j-jobs #1115

lgannoaa · 2022-11-10T15:09:31Z

Description
This PR is to update the archive jobs using arc.sh, j-job, ex-script approach similar style to the package format.

The rocoto workflow task job script arch.sh earc.sh in $HOMEgfs/jobs/rocoto are updated to adopt workflow style.
Introduce archive J-Jobs in $HOMEgfs/jobs with J-Job style.
Introduce archive ex-scripts in $HOMEgfs/scripts to perform task works that is in-line with the rest of the workflow.

Fixes #1051
Refs: #720

How Has This Been Tested?
C192C96 cycled run tested on Cactus

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings

Include script, JJob, and utility for GDAS, ENKFGDAS, and GFS archive jobs Detailed information can be found in github issue NOAA-EMC#1051 NOAA-EMC#1051

Move JJob and scripts into HOMEgfs level

…into feature/ecf-archive

Github issue NOAA-EMC#1051

github-advanced-security

shellcheck found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

aerorahul

Overall the changes are what they are supposed to do.
Pending a few comments to move specific lines from J-Job to exscript, this should be ready for develop.

jobs/JENKFGDAS_ARCHIVE

aerorahul · 2022-11-14T14:44:53Z

jobs/JENKFGDAS_ARCHIVE

+export EARCINC_CYC=${ARCH_CYC}
+export EARCICS_CYC=$((ARCH_CYC-assim_freq))
+if [ ${EARCICS_CYC} -lt 0 ]; then
+  export EARCICS_CYC=$((EARCICS_CYC+24))
+fi


Please move these to exgdas_enkf_earc.sh and drop the export

aerorahul · 2022-11-14T14:45:32Z

jobs/JENKFGDAS_ARCHIVE

+# EnKF update in GFS, GDAS or both
+export CDUMP_ENKF=$(echo ${EUPD_CYC:-"gdas"} | tr a-z A-Z)
+
+export ARCH_LIST="${ROTDIR}/enkf${CDUMP}.${PDY}/${cyc}/${COMPONENT}/earc${ENSGRP}"


This belongs to exgdas_enkf_earc.sh and is there.

aerorahul · 2022-11-14T14:46:12Z

scripts/exgdas_enkf_earc.sh

+###############################################################
+## Abstract:
+## Ensemble archive driver script
+## RUN_ENVIR : runtime environment (emc | nco)
+## HOMEgfs   : /full/path/to/workflow
+## EXPDIR : /full/path/to/config/files
+## CDATE  : current analysis date (YYYYMMDDHH)
+## PDY    : current date (YYYYMMDD)
+## cyc    : current cycle (HH)
+## CDUMP  : cycle name (gdas / gfs)
+## ENSGRP : ensemble sub-group to archive (0, 1, 2, ...)
+###############################################################


You may drop these lines.

aerorahul · 2022-11-14T14:49:48Z

jobs/JENKFGDAS_ARCHIVE

+#############################
+export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
+configs="base earc"
+config_path=${EXPDIR:-${NWROOT}/gfs.${gfs_ver}/parm/config}


Suggested change

config_path=${EXPDIR:-${NWROOT}/gfs.${gfs_ver}/parm/config}

config_path=${EXPDIR:-${PACKAGEROOT}/gfs.${gfs_ver}/parm/config}

aerorahul · 2022-11-14T14:54:28Z

jobs/JGLOBAL_ARCHIVE

+#############################################
+# Source workflow level common utilities
+#############################################
+#### This utility is under development as of 20221020
+####   will not use it at this time
+#### source "${HOMEgfs}/ush/file_utils.sh"


Please delete.
These are being sourced and used in exglobal_archive.sh

aerorahul · 2022-11-14T14:55:30Z

jobs/JGLOBAL_ARCHIVE

+export ARCHINC_CYC=$ARCH_CYC
+export ARCHICS_CYC=$((ARCH_CYC-assim_freq))
+if [ $ARCHICS_CYC -lt 0 ]; then
+    ARCHICS_CYC=$((ARCHICS_CYC+24))
+fi
+
+# CURRENT CYCLE
+export APREFIX="${CDUMP}.t${cyc}z."
+export ASUFFIX=${ASUFFIX:-$SUFFIX}
+
+if [ $ASUFFIX = ".nc" ]; then
+   format="netcdf"
+else
+   format="nemsio"
+fi


Please move these to exglobal_archive.sh and drop the export

scripts/exglobal_archive.sh

lgannoaa · 2022-11-14T16:32:09Z

@aerorahul, recommended changes have been made.

aerorahul

Changes look good.
Pending fixing the warnings, and a successful test and review from the other reviewers, this can be merged.

WalterKolczynski-NOAA

I'm assuming the code in the new exscripts is just what was in the rocoto script before, so I'm not going to pick it apart here.

jobs/JENKFGDAS_ARCHIVE

jobs/JGLOBAL_ARCHIVE

WalterKolczynski-NOAA

Approved pending linter fixes and final testing.

lgannoaa · 2022-11-14T21:28:58Z

Tested in Cactus with C192C96 cycled run. Job completed.

WalterKolczynski-NOAA · 2022-11-16T15:31:22Z

Tested in Cactus with C192C96 cycled run. Job completed.

Please make sure you also test coupled, since it creates a different set of archive files. Coupled isn't fully working on WCOSS2 yet, so you'll have to use an RDHPCS machine.

aerorahul · 2022-11-16T22:27:09Z

@WalterKolczynski-NOAA Some of the shellcheck suggestions are likely out of scope from this PR e.g. tr since we are just moving code around. Do we need to address those as well?

WalterKolczynski-NOAA · 2022-11-17T19:18:55Z

@WalterKolczynski-NOAA Some of the shellcheck suggestions are likely out of scope from this PR e.g. tr since we are just moving code around. Do we need to address those as well?

Since this is shifting code around, just leave them. We'll take care of them later.

lgannoaa · 2022-11-18T14:20:38Z

The C384 ATM fcst only test on Hera completed. The Hera space is too low for ATMW cycled test. Will proceed that test on wcoss.
The COM for C384 ATM fcst only test on Hera will be removed at noon today 11/18/22 to release some space.

lgannoaa · 2022-11-21T19:10:18Z

C768C384 ATMW couple cycled test executed on wcoss2. All jobs completed as expected.
@aerorahul @WalterKolczynski-NOAA let me know if you have any questions.

WalterKolczynski-NOAA · 2022-11-21T19:14:51Z

Still needs a full coupled S2SWA test. It will have to be run on an RDHPCS machine because it won't work on WCOSS2 yet. Workflow will automatically get the (C384) ICs for you. Use 2013040100.

lgannoaa · 2022-11-22T14:46:04Z

Full coupled S2SWA fcst only test on Orion is done with gfsarch job status completed.
log: /work/noaa/stmp/lgan/comrot/s2swa_arch_orion/logs/2013040100

* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)

lgannoaa added 14 commits October 7, 2022 16:36

GFS archive jobs designed to increase GFS workflow stability.

77e9aeb

Include script, JJob, and utility for GDAS, ENKFGDAS, and GFS archive jobs Detailed information can be found in github issue NOAA-EMC#1051 NOAA-EMC#1051

Remove ecflow components

5dfb5e0

Move JJob and scripts into HOMEgfs level

Merge branch 'develop' of https://github.com/NOAA-EMC/global-workflow …

0a40e5f

…into feature/ecf-archive

Init commit

f80ee72

Add J-Job and ex script for archive job development

838ecd7

pull Merge upstream

25846f7

Merge branch 'develop' of https://github.com/NOAA-EMC/global-workflow …

e729e01

…into feature/ecf-archive

Enable GDAS and GFS archive. Jobs tested on Cactus with cycled C192/C96.

7a6b62d

Github issue NOAA-EMC#1051

Create a version of ecflow archive jobs in rocoto workflow

103b6a2

Code clean up for PR preparation

c29bca3

Restore a missing file from develop

9fc35a9

Create a version that only have jjob and ex-script updates

bfe11c6

Revert local change on vrfy script

9930520

Revert local fix to build issue

8c045ce

lgannoaa requested a review from aerorahul November 10, 2022 15:09

github-advanced-security bot found potential problems Nov 10, 2022

View reviewed changes

aerorahul marked this pull request as ready for review November 10, 2022 18:53

lgannoaa self-assigned this Nov 10, 2022

JENKFGDAS_ARCHIVE apply filter

ed8bd67

aerorahul requested a review from WalterKolczynski-NOAA November 14, 2022 14:42

aerorahul reviewed Nov 14, 2022

View reviewed changes

lgannoaa added 2 commits November 14, 2022 16:25

Rearrange some script statement between jjob and ex-script

9715c78

Remove extra statement in JENKFGDAS_ARCHIVE

b97d453

aerorahul approved these changes Nov 14, 2022

View reviewed changes

Putting braces around variable

686a63d

WalterKolczynski-NOAA requested changes Nov 14, 2022

View reviewed changes

jobs/JENKFGDAS_ARCHIVE Outdated Show resolved Hide resolved

jobs/JGLOBAL_ARCHIVE Outdated Show resolved Hide resolved

remove date statement from jjobs

e802183

WalterKolczynski-NOAA approved these changes Nov 14, 2022

View reviewed changes

lgannoaa added 5 commits November 16, 2022 19:57

Apply shellcheck 2004,2086,2166,2248,2250

d286dce

Apply shellcheck 2004,2086,2166,2248,2250 to rocoto jobs

9d54a70

Apply shell check sc2004 to exgdas_enkf_earc.sh and exglobal_archive.sh

de26d30

Update syntex

11b1718

If statement update for exgdas_enkf_earc

823f5f6

lgannoaa mentioned this pull request Nov 22, 2022

Merge ecflow workflow archive jobs back to global workflow #1051

Closed

WalterKolczynski-NOAA changed the title ~~Archive job updates~~ Convert archive jobs to proper j-jobs Nov 22, 2022

WalterKolczynski-NOAA merged commit 5df144d into NOAA-EMC:develop Nov 22, 2022

aerorahul mentioned this pull request Feb 28, 2023

Creating J jobs for archive #720

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert archive jobs to proper j-jobs #1115

Convert archive jobs to proper j-jobs #1115

lgannoaa commented Nov 10, 2022 •

edited by WalterKolczynski-NOAA

Loading

github-advanced-security bot left a comment

aerorahul left a comment

aerorahul Nov 14, 2022

aerorahul Nov 14, 2022

aerorahul Nov 14, 2022

aerorahul Nov 14, 2022

aerorahul Nov 14, 2022

aerorahul Nov 14, 2022

lgannoaa commented Nov 14, 2022

aerorahul left a comment

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA left a comment

lgannoaa commented Nov 14, 2022

WalterKolczynski-NOAA commented Nov 16, 2022

aerorahul commented Nov 16, 2022

WalterKolczynski-NOAA commented Nov 17, 2022

lgannoaa commented Nov 18, 2022 •

edited

Loading

lgannoaa commented Nov 21, 2022

WalterKolczynski-NOAA commented Nov 21, 2022

lgannoaa commented Nov 22, 2022 •

edited

Loading

	config_path=${EXPDIR:-${NWROOT}/gfs.${gfs_ver}/parm/config}
	config_path=${EXPDIR:-${PACKAGEROOT}/gfs.${gfs_ver}/parm/config}

Convert archive jobs to proper j-jobs #1115

Convert archive jobs to proper j-jobs #1115

Conversation

lgannoaa commented Nov 10, 2022 • edited by WalterKolczynski-NOAA Loading

github-advanced-security bot left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

aerorahul Nov 14, 2022

Choose a reason for hiding this comment

lgannoaa commented Nov 14, 2022

aerorahul left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

lgannoaa commented Nov 14, 2022

WalterKolczynski-NOAA commented Nov 16, 2022

aerorahul commented Nov 16, 2022

WalterKolczynski-NOAA commented Nov 17, 2022

lgannoaa commented Nov 18, 2022 • edited Loading

lgannoaa commented Nov 21, 2022

WalterKolczynski-NOAA commented Nov 21, 2022

lgannoaa commented Nov 22, 2022 • edited Loading

lgannoaa commented Nov 10, 2022 •

edited by WalterKolczynski-NOAA

Loading

lgannoaa commented Nov 18, 2022 •

edited

Loading

lgannoaa commented Nov 22, 2022 •

edited

Loading