First step towards making j-jobs consistent in use from ecflow and rocoto #1120

aerorahul · 2022-11-16T15:53:06Z

Description

Several jobs run in development mode do not adhere to EE2 standards. Sometimes these development jobs do get used for operations at which time there is a rush in getting them to be standardized.
These jobs are also used to run parallel retrospectives and near-real time parallels.

This work is performed with GFSv17 and GEFSv13 as the target implementation.

This PR attempts to standardize the use of several variables as defined in the NCO EE2 standards document.
Specifically;
The following are defined in the job-card:
DATAROOT
cyc
HOMEgfs
job
jobid=$job/$$ or jobid=$PBS_JOBNAME/$PBS_JOBID depending on Rocoto/Ops

The following are defined in the j-job:
pid
DATA=DATAROOT/jobid
cycle

This PR also fixes a bug in exgfs_atmos_post.sh that is only encountered when WAFSF=YES and in the gdaspostanl job. An overzealous linter added extra white-space and put quotes around an argument for wgrib2.

Type of change

New feature (non-breaking change which adds functionality)

It is not intended to break any frequently used existing features, however testing of several features e.g. WAFS jobs, BUFRSND jobs are untested in develop. It is unlikely these jobs run in develop to begin with and fixing them will be done in a future PR.

Several of the development jobs e.g. vrfy, metp, prep, ocnpost, arch, earc do not follow EE2 standards.
Testing was done to ensure these jobs run without failure, however extended testing is necessary to validate these tests as some of these jobs require extended data to be generated.

Future additions of jobs/scripts will be checked for EE2 compliance, regardless of if it is used in development or operations.

How Has This Been Tested?
All tests were performed on Orion.

Cycled test for two and half cycles
Single prototype S2SW forecast-only test case

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
[?] New and existing tests pass with my changes
Any dependent changes have been merged and published

…g.sh

…o fix that

… fail

…ion, and yet the task returns with an exit status 0.

@YaliMao-NOAA

…is not available on Orion. They need to be tested on wcoss2. asking @YaliMao-NOAA would be help

…un on Orion.

jobs/JGDAS_ATMOS_GEMPAK

-if [ "$KEEPDATA" != "YES" ] ; then
-  rm -rf $DATA
+if [ "${KEEPDATA}" != "YES" ] ; then
+  rm -rf ${DATA}


jobs/rocoto/vrfy.sh

@@ -184,7 +192,8 @@

 ###############################################################
 # Force Exit out cleanly
-if [ ${KEEPDATA:-"NO"} = "NO" ] ; then rm -rf "${DATAROOT}" ; fi
+cd ${DATAROOT}
+if [ ${KEEPDATA:-"NO"} = "NO" ] ; then rm -rf ${DATA} ; fi


jobs/rocoto/vrfy.sh

@@ -184,7 +192,8 @@

 ###############################################################
 # Force Exit out cleanly
-if [ ${KEEPDATA:-"NO"} = "NO" ] ; then rm -rf "${DATAROOT}" ; fi
+cd ${DATAROOT}
+if [ ${KEEPDATA:-"NO"} = "NO" ] ; then rm -rf ${DATA} ; fi


jobs/JGDAS_ATMOS_GEMPAK

jobs/JGDAS_ATMOS_VERFRAD

-if [ ${KEEPDATA} = NO ] ; then 
-  rm -rf $RAD_DATA_IN
+cd ${DATAROOT}
+if [ ${KEEPDATA} = NO ] ; then


jobs/JGDAS_ATMOS_VERFRAD

-  rm -rf $RAD_DATA_IN
+cd ${DATAROOT}
+if [ ${KEEPDATA} = NO ] ; then
+  rm -rf ${RAD_DATA_IN}


jobs/JGDAS_ATMOS_GEMPAK

KateFriedman-NOAA · 2022-11-29T20:47:43Z

jobs/JGDAS_ATMOS_VERFRAD

+mkdir -p ${RAD_DATA_IN}
+cd ${RAD_DATA_IN}


Can we just use ${DATA} instead of ${RAD_DATA_IN}?

These lines are technically not necessary, as we are already in DATA from lines 12, 13.

Also, I did not add them, just ran them through the linter.

KateFriedman-NOAA · 2022-11-29T20:49:06Z

jobs/JGDAS_ATMOS_VERFRAD

+#  gfs v15 which doesn't have
+#  a $COMPONENT in output path
 ################################
 if [[ ! -d ${COMIN} ]]; then
   export COMIN=${COM_IN}/${RUN}.${PDY}/${cyc}
 fi


Retire this?

I can't tell.
Definitely not part of this PR.

WalterKolczynski-NOAA

Not too much. A couple of the standardization things might be mooted by moving to a unified script for the first section of j-jobs, if you want to defer those.

jobs/JGDAS_ATMOS_ANALYSIS_DIAG

jobs/JGDAS_ATMOS_GEMPAK

jobs/JGDAS_ATMOS_GLDAS

jobs/JGDAS_ATMOS_VERFOZN

WalterKolczynski-NOAA · 2022-11-29T18:34:20Z

jobs/JGDAS_GLOBAL_ATMOS_ANALYSIS_POST

-cd $DATA || (echo "$DATA does not exist. ABORT!"; exit 1)
+export DATA=${DATA:-${DATAROOT}/${jobid}}
+mkdir -p ${DATA}
+cd ${DATA} || (echo "${DATA} does not exist. ABORT!"; exit 1)


We should use this as the prototype for all the other cd ${DATA} so it is consistent. Though maybe change message to "FATAL: Could not change current directory to ${DATA}".

jobs/JGFS_ATMOS_FSU_GENESIS

jobs/JGFS_ATMOS_PGRB2_SPEC_NPOESS

scripts/exglobal_atmos_analysis.sh

workflow/applications.py

jobs/JGDAS_ATMOS_GLDAS

-    # shellcheck disable=SC1090-SC1091
-    . "${config_path}/config.${config}"
-    # shellcheck disable=
+    . ${EXPDIR}/config.${config}


jobs/JGLOBAL_ATMOS_POST

+export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
+configs="base post"
+for config in ${configs}; do
+    . ${EXPDIR}/config.${config}


jobs/JENKFGDAS_ARCHIVE

    status=$?
    [[ ${status} -ne 0 ]] && exit "${status}"
 done

 ##########################################
 # Source machine runtime environment
 ##########################################
-. "${HOMEgfs}"/env/"${machine}".env earc
+. ${HOMEgfs}/env/${machine}.env earc


jobs/JENKFGDAS_ARCHIVE


 ###############################################################
 # Run archive script
 ###############################################################

-"${SCRgfs}"/exgdas_enkf_earc.sh
+${SCRgfs}/exgdas_enkf_earc.sh


jobs/JGLOBAL_ARCHIVE

 for config in ${configs}; do
-    . "${config_path}"/config."${config}"
+    . ${EXPDIR}/config.${config}


jobs/JENKFGDAS_ARCHIVE

    status=$?
    [[ ${status} -ne 0 ]] && exit "${status}"
 done

 ##########################################
 # Source machine runtime environment
 ##########################################
-. "${HOMEgfs}"/env/"${machine}".env earc
+. ${HOMEgfs}/env/${machine}.env earc


jobs/JGDAS_ATMOS_GLDAS

-    # shellcheck disable=SC1090-SC1091
-    . "${config_path}/config.${config}"
-    # shellcheck disable=
+    . ${EXPDIR}/config.${config}


jobs/JGLOBAL_ARCHIVE

 for config in ${configs}; do
-    . "${config_path}"/config."${config}"
+    . ${EXPDIR}/config.${config}


jobs/JGLOBAL_ATMOS_POST

+export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
+configs="base post"
+for config in ${configs}; do
+    . ${EXPDIR}/config.${config}


The new marine DA jobs were added (PR #1134) roughly concurrently with the refactoring of the j-job scripts (PR #1120), and thus didn't pick up the changes there. These new jobs are now updated.

* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)

aerorahul and others added 30 commits October 11, 2022 17:15

wip. fixes for forecast jobs

1119a68

Merge branch 'develop' into feature/ecf-rocoto-jobs

6d93c4c

fix echgres

d73bb88

fix epos

72b3945

fix post

4bf4f97

fix vrfy

4a452a8

fix anal

3bf4e58

fix eobs, sfcanl. cycle variable should be set in j-job

d783c31

cycle variable is defined in j-job not the job-card

b8a9292

fix analcalc

e140e49

COMPONENT variable is/cannot be set from above in a JJob

b962566

fix ediag

ad67dca

fix ecen, esfc and eupd

703cad0

bring GSI monitoring jobs up

fd26cb3

jobs/JGDAS_ENKF_INNOVATE_OBS does not exist. There is no need for eom…

7c71198

…g.sh

fix jedi tasks

a5aeb17

update ATMOS_GEMPAK without testing

1706714

update to wave jobs. some wave jobs do not source config.base. need t…

e55869f

…o fix that

fix some rocoto/wave jobs that call jjobs

fc8bd5c

fix metp

bffbf23

WAFS toggle in config.base is WAFSF and not DO_WAFS

e9178c4

remove trailing whitespace and incorrect quotes that caused wgrib2 to…

bca4e0f

… fail

fix wafs. There are FATAL_ERRORs noticed in these jobs when run on Or…

0b457b3

…ion, and yet the task returns with an exit status 0.

fix wafsgrib2 and wafsgrib20p25. These jobs depend on some data that …

f819a39

…is not available on Orion. They need to be tested on wcoss2. asking @YaliMao-NOAA would be help

fix wafsblending jobs. They depend on wafsgrib2 jobs which were not r…

be74f74

…un on Orion.

fix postsnd

87322f9

fix ocnpost

23fbb5a

fix cyc, jobid and DATA

c7a3db5

Merge branch 'develop' into feature/ecf-rocoto-jobs

be4c310

Merge branch 'develop' into feature/ecf-rocoto-jobs

90766e0

run shellcheck with SC2250 on these scripts

13102c0

aerorahul requested review from WalterKolczynski-NOAA and KateFriedman-NOAA November 16, 2022 15:53

Merge branch 'develop' into feature/ecf-rocoto-jobs

396b355

github-advanced-security bot found potential problems Nov 16, 2022

View reviewed changes

aerorahul marked this pull request as ready for review November 29, 2022 15:10

aerorahul added 2 commits November 29, 2022 12:44

Merge branch 'develop' into feature/ecf-rocoto-jobs

0639978

Merge branch 'develop' into feature/ecf-rocoto-jobs

fcf5600

KateFriedman-NOAA reviewed Nov 29, 2022

View reviewed changes

jobs/JGDAS_ATMOS_GEMPAK Outdated Show resolved Hide resolved

KateFriedman-NOAA reviewed Nov 29, 2022

View reviewed changes

WalterKolczynski-NOAA requested changes Nov 29, 2022

View reviewed changes

remove references to config_path

3e26d0e

github-advanced-security bot found potential problems Dec 1, 2022

View reviewed changes

correct typo

63db45b

aerorahul requested review from WalterKolczynski-NOAA and KateFriedman-NOAA December 5, 2022 22:29

WalterKolczynski-NOAA approved these changes Dec 5, 2022

View reviewed changes

KateFriedman-NOAA approved these changes Dec 6, 2022

View reviewed changes

WalterKolczynski-NOAA merged commit 6ab858e into NOAA-EMC:develop Dec 6, 2022

aerorahul deleted the feature/ecf-rocoto-jobs branch December 6, 2022 16:06

aerorahul mentioned this pull request Dec 6, 2022

Update marine DA j-jobs to new format #1149

Merged

7 tasks

aerorahul mentioned this pull request Dec 23, 2022

Bring j-jobs up to EE2 compliance. #1069

Open

DavidHuber-NOAA mentioned this pull request Apr 23, 2024

Remove the eomg job #2525

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

aerorahul commented Nov 16, 2022

KateFriedman-NOAA Nov 29, 2022

aerorahul Dec 1, 2022

aerorahul Dec 1, 2022

KateFriedman-NOAA Nov 29, 2022

aerorahul Dec 1, 2022

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Nov 29, 2022

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

Conversation

aerorahul commented Nov 16, 2022

KateFriedman-NOAA Nov 29, 2022

Choose a reason for hiding this comment

aerorahul Dec 1, 2022

Choose a reason for hiding this comment

aerorahul Dec 1, 2022

Choose a reason for hiding this comment

KateFriedman-NOAA Nov 29, 2022

Choose a reason for hiding this comment

aerorahul Dec 1, 2022

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Nov 29, 2022

Choose a reason for hiding this comment