Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

Merged
merged 36 commits into from
Dec 6, 2022
Merged

First step towards making j-jobs consistent in use from ecflow and rocoto #1120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
1119a68
wip. fixes for forecast jobs
aerorahul Oct 11, 2022
6d93c4c
Merge branch 'develop' into feature/ecf-rocoto-jobs
aerorahul Oct 12, 2022
d73bb88
fix echgres
aerorahul Oct 12, 2022
72b3945
fix epos
aerorahul Oct 12, 2022
4bf4f97
fix post
aerorahul Oct 12, 2022
4a452a8
fix vrfy
aerorahul Oct 12, 2022
3bf4e58
fix anal
aerorahul Oct 12, 2022
d783c31
fix eobs, sfcanl. cycle variable should be set in j-job
aerorahul Oct 12, 2022
b8a9292
cycle variable is defined in j-job not the job-card
aerorahul Oct 12, 2022
e140e49
fix analcalc
aerorahul Oct 12, 2022
b962566
COMPONENT variable is/cannot be set from above in a JJob
aerorahul Oct 12, 2022
ad67dca
fix ediag
aerorahul Oct 12, 2022
703cad0
fix ecen, esfc and eupd
aerorahul Oct 12, 2022
fd26cb3
bring GSI monitoring jobs up
aerorahul Oct 12, 2022
7c71198
jobs/JGDAS_ENKF_INNOVATE_OBS does not exist. There is no need for eo…
aerorahul Oct 12, 2022
a5aeb17
fix jedi tasks
aerorahul Oct 12, 2022
1706714
update ATMOS_GEMPAK without testing
aerorahul Oct 12, 2022
e55869f
update to wave jobs. some wave jobs do not source config.base. need …
aerorahul Oct 12, 2022
fc8bd5c
fix some rocoto/wave jobs that call jjobs
aerorahul Oct 13, 2022
bffbf23
fix metp
aerorahul Oct 13, 2022
e9178c4
WAFS toggle in config.base is WAFSF and not DO_WAFS
aerorahul Oct 13, 2022
bca4e0f
remove trailing whitespace and incorrect quotes that caused wgrib2 to…
aerorahul Oct 14, 2022
0b457b3
fix wafs. There are FATAL_ERRORs noticed in these jobs when run on Or…
aerorahul Oct 14, 2022
f819a39
fix wafsgrib2 and wafsgrib20p25. These jobs depend on some data that …
aerorahul Oct 14, 2022
be74f74
fix wafsblending jobs. They depend on wafsgrib2 jobs which were not …
aerorahul Oct 14, 2022
87322f9
fix postsnd
aerorahul Oct 17, 2022
23fbb5a
fix ocnpost
aerorahul Oct 18, 2022
c7a3db5
fix cyc, jobid and DATA
aerorahul Oct 18, 2022
be4c310
Merge branch 'develop' into feature/ecf-rocoto-jobs
aerorahul Oct 18, 2022
90766e0
Merge branch 'develop' into feature/ecf-rocoto-jobs
Nov 14, 2022
13102c0
run shellcheck with SC2250 on these scripts
aerorahul Nov 16, 2022
396b355
Merge branch 'develop' into feature/ecf-rocoto-jobs
aerorahul Nov 16, 2022
0639978
Merge branch 'develop' into feature/ecf-rocoto-jobs
aerorahul Nov 29, 2022
fcf5600
Merge branch 'develop' into feature/ecf-rocoto-jobs
aerorahul Nov 29, 2022
3e26d0e
remove references to config_path
aerorahul Dec 1, 2022
63db45b
correct typo
aerorahul Dec 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,6 @@ export NTHSTACK=1024000000
ulimit -s unlimited
ulimit -a

export job=${PBS_JOBNAME:-${step}}
export jobid=${job}.${PBS_JOBID:-$$}


if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then

nth_max=$((npe_node_max / npe_node_prep))
Expand Down
3 changes: 0 additions & 3 deletions env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ export NTHSTACK=1024000000
ulimit -s unlimited
ulimit -a

export job=${PBS_JOBNAME:-${step}}
export jobid=${job}.${PBS_JOBID:-$$}

if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then

nth_max=$((npe_node_max / npe_node_prep))
Expand Down
3 changes: 0 additions & 3 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,6 @@ export NTHSTACK=1024000000
ulimit -s unlimited
ulimit -a

export job=${PBS_JOBNAME:-${step}}
export jobid=${job}.${PBS_JOBID:-$$}

if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then

nth_max=$((npe_node_max / npe_node_prep))
Expand Down
6 changes: 1 addition & 5 deletions env/S4.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash -x
#!/usr/bin/env bash

if [[ $# -ne 1 ]]; then

Expand Down Expand Up @@ -30,10 +30,6 @@ export NTHSTACK=1024000000
ulimit -s unlimited
ulimit -a

export job=${PBS_JOBNAME:-${step}}
export jobid=${job}.${PBS_JOBID:-$$}


if [[ ${step} = "prep" || ${step} = "prepbufr" ]]; then

npe_node_prep=${npe_node_prep:-${npe_node_max}}
Expand Down
3 changes: 0 additions & 3 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,6 @@ export mpmd_opt="--cpu-bind verbose,core cfp"

export npe_node_max=128

export job=${PBS_JOBNAME:-${step}}
export jobid=${job}.${PBS_JOBID:-$$}

if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then

nth_max=$((npe_node_max / npe_node_prep))
Expand Down
9 changes: 4 additions & 5 deletions jobs/JENKFGDAS_ARCHIVE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,16 @@ source "${HOMEgfs}/ush/preamble.sh"
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base earc"
config_path=${EXPDIR:-${PACKAGEROOT}/gfs.${gfs_ver}/parm/config}
for config in ${configs}; do
. "${config_path}"/config."${config}"
. "${EXPDIR}"/config."${config}"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

##########################################
# Source machine runtime environment
##########################################
. "${HOMEgfs}"/env/"${machine}".env earc
. ${HOMEgfs}/env/${machine}.env earc

Check notice

Code scanning / shellcheck

Double quote to prevent globbing and word splitting.

Double quote to prevent globbing and word splitting.

Check notice

Code scanning / shellcheck

Double quote to prevent globbing and word splitting.

Double quote to prevent globbing and word splitting.
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"

Expand Down Expand Up @@ -50,13 +49,13 @@ export pgmerr=errfile
##############################################
export CDATE=${CDATE:-${PDY}${cyc}}
export CDUMP=${CDUMP:-${RUN:-"gdas"}}
export COMPONENT=${COMPONENT:-atmos}
export COMPONENT="atmos"

###############################################################
# Run archive script
###############################################################

"${SCRgfs}"/exgdas_enkf_earc.sh
${SCRgfs}/exgdas_enkf_earc.sh

Check notice

Code scanning / shellcheck

Double quote to prevent globbing and word splitting.

Double quote to prevent globbing and word splitting.
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"

Expand Down
119 changes: 57 additions & 62 deletions jobs/JGDAS_ATMOS_ANALYSIS_DIAG
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,13 @@
#! /usr/bin/env bash

source "$HOMEgfs/ush/preamble.sh"

export RUN_ENVIR=${RUN_ENVIR:-"nco"}

#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
configs="base anal analdiag"
config_path=${EXPDIR:-$PACKAGEROOT/gfs.${gfs_ver}/parm/config}
for config in $configs; do
. $config_path/config.$config
status=$?
[[ $status -ne 0 ]] && exit $status
done


##########################################
# Source machine runtime environment
##########################################
. $HOMEgfs/env/${machine}.env anal
status=$?
[[ $status -ne 0 ]] && exit $status

source "${HOMEgfs}/ush/preamble.sh"

##############################################
# Obtain unique process id (pid) and make temp directory
# make temp directory
##############################################
export pid=${pid:-$$}
export outid=${outid:-"LL$job"}

export DATA=${DATA:-${DATAROOT}/${jobid:?}}
mkdir -p $DATA
cd $DATA
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}


##############################################
Expand All @@ -47,72 +21,93 @@ setpdy.sh
##############################################
# Determine Job Output Name on System
##############################################
export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base anal analdiag"
for config in ${configs}; do
. ${EXPDIR}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
done


##########################################
# Source machine runtime environment
##########################################
. ${HOMEgfs}/env/${machine}.env anal
status=$?
[[ ${status} -ne 0 ]] && exit ${status}


##############################################
# Set variables used in the script
##############################################
export CDATE=${CDATE:-${PDY}${cyc}}
export CDUMP=${CDUMP:-${RUN:-"gfs"}}
export COMPONENT=${COMPONENT:-atmos}
export DO_CALC_ANALYSIS=${DO_CALC_ANALYSIS:-"YES"}
export COMPONENT="atmos"
export DO_CALC_ANALYSIS=${DO_CALC_ANALYSIS:-"YES"}


##############################################
# Begin JOB SPECIFIC work
##############################################

GDATE=$($NDATE -$assim_freq $CDATE)
gPDY=$(echo $GDATE | cut -c1-8)
gcyc=$(echo $GDATE | cut -c9-10)
GDATE=$(${NDATE} -${assim_freq} ${CDATE})
gPDY=$(echo ${GDATE} | cut -c1-8)
gcyc=$(echo ${GDATE} | cut -c9-10)
GDUMP=${GDUMP:-"gdas"}

export OPREFIX="${CDUMP}.t${cyc}z."
export GPREFIX="${GDUMP}.t${gcyc}z."
export APREFIX="${CDUMP}.t${cyc}z."
export GSUFFIX=${GSUFFIX:-$SUFFIX}
export ASUFFIX=${ASUFFIX:-$SUFFIX}
export GSUFFIX=${GSUFFIX:-${SUFFIX}}
export ASUFFIX=${ASUFFIX:-${SUFFIX}}


if [ $RUN_ENVIR = "nco" -o ${ROTDIR_DUMP:-NO} = "YES" ]; then
export COMIN=${COMIN:-$ROTDIR/$RUN.$PDY/$cyc/$COMPONENT}
export COMOUT=${COMOUT:-$ROTDIR/$RUN.$PDY/$cyc/$COMPONENT}
export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$RUN.$PDY/$cyc/$COMPONENT}
export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/$GDUMP.$gPDY/$gcyc/$COMPONENT}
if [ ${RUN_ENVIR} = "nco" -o ${ROTDIR_DUMP:-NO} = "YES" ]; then
export COMIN=${COMIN:-${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}}
export COMOUT=${COMOUT:-${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}}
export COMIN_OBS=${COMIN_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/${RUN}.${PDY}/${cyc}/${COMPONENT}}
export COMIN_GES_OBS=${COMIN_GES_OBS:-$(compath.py ${envir}/obsproc/${obsproc_ver})/${GDUMP}.${gPDY}/${gcyc}/${COMPONENT}}
else
export COMOUT="$ROTDIR/$CDUMP.$PDY/$cyc/$COMPONENT"
export COMIN_OBS="$DMPDIR/$CDUMP.$PDY/$cyc/$COMPONENT"
export COMIN_GES_OBS="$DMPDIR/$GDUMP.$gPDY/$gcyc/$COMPONENT"
export COMOUT="${ROTDIR}/${CDUMP}.${PDY}/${cyc}/${COMPONENT}"
export COMIN_OBS="${DMPDIR}/${CDUMP}.${PDY}/${cyc}/${COMPONENT}"
export COMIN_GES_OBS="${DMPDIR}/${GDUMP}.${gPDY}/${gcyc}/${COMPONENT}"
fi
mkdir -m 775 -p $COMOUT
mkdir -m 775 -p ${COMOUT}
# COMIN_GES and COMIN_GES_ENS are used in script
export COMIN_GES="$ROTDIR/$GDUMP.$gPDY/$gcyc/$COMPONENT"
export COMIN_GES_ENS="$ROTDIR/enkfgdas.$gPDY/$gcyc/$COMPONENT"
export COMIN_GES="${ROTDIR}/${GDUMP}.${gPDY}/${gcyc}/${COMPONENT}"
export COMIN_GES_ENS="${ROTDIR}/enkfgdas.${gPDY}/${gcyc}/${COMPONENT}"


export ATMGES="$COMIN_GES/${GPREFIX}atmf006${GSUFFIX}"
if [ ! -f $ATMGES ]; then
echo "FATAL ERROR: FILE MISSING: ATMGES = $ATMGES"
export ATMGES="${COMIN_GES}/${GPREFIX}atmf006${GSUFFIX}"
if [ ! -f ${ATMGES} ]; then
echo "FATAL ERROR: FILE MISSING: ATMGES = ${ATMGES}"
exit 1
fi


if [ $DOHYBVAR = "YES" ]; then
export ATMGES_ENSMEAN="$COMIN_GES_ENS/${GPREFIX}atmf006.ensmean$GSUFFIX"
if [ ! -f $ATMGES_ENSMEAN ]; then
echo "FATAL ERROR: FILE MISSING: ATMGES_ENSMEAN = $ATMGES_ENSMEAN"
if [ ${DOHYBVAR} = "YES" ]; then
export ATMGES_ENSMEAN="${COMIN_GES_ENS}/${GPREFIX}atmf006.ensmean${GSUFFIX}"
if [ ! -f ${ATMGES_ENSMEAN} ]; then
echo "FATAL ERROR: FILE MISSING: ATMGES_ENSMEAN = ${ATMGES_ENSMEAN}"
exit 2
fi
fi


###############################################################
# Run relevant script
${ANALDIAGSH:-$SCRgfs/exglobal_diag.sh}
${ANALDIAGSH:-${SCRgfs}/exglobal_diag.sh}
status=$?
[[ $status -ne 0 ]] && exit $status
[[ ${status} -ne 0 ]] && exit ${status}


##############################################
Expand All @@ -122,15 +117,15 @@ status=$?
##############################################
# Final processing
##############################################
if [ -e "$pgmout" ] ; then
cat $pgmout
if [ -e "${pgmout}" ] ; then
cat ${pgmout}
fi

##########################################
# Remove the Temporary working directory
##########################################
cd $DATAROOT
[[ $KEEPDATA = "NO" ]] && rm -rf $DATA
cd ${DATAROOT}
[[ ${KEEPDATA} = "NO" ]] && rm -rf ${DATA}


exit 0
Loading