Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[develop] Add process lightning task #644

Merged
merged 20 commits into from
Apr 14, 2023
Merged

[develop] Add process lightning task #644

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a4fd651
Added rrfs_utils to the cmake settings
Mar 6, 2023
3887c34
Added jobs and exregional lightning scripts
Mar 7, 2023
cb5aa41
Added lightning parm
Mar 7, 2023
2fb34e8
did some cleanup
Mar 15, 2023
75aa2f2
added lightning wrapper
Mar 15, 2023
fa884bb
added rrfs lightning vars
Mar 15, 2023
4cd5472
add fix gsi path
Mar 15, 2023
08786ff
Merge branch 'ufs-community:develop' into feature/rrfs-lightning
EdwardSnyder-NOAA Mar 15, 2023
8f16d66
Merge branch 'feature/rrfs-lightning' of https://github.com/EdwardSny…
Mar 15, 2023
0f781e6
removed fixgsi dir
Mar 15, 2023
fc0bcad
updated path to predefined grid for fv3_grid_spec
Mar 22, 2023
aa37c87
added lightning root and fixgsi vars
Mar 22, 2023
ce82fd1
updated DOMAIN_PREGEN_BASEDIR to point to EPIC location
Mar 22, 2023
b5cf64d
removed unneeded date paths
Mar 27, 2023
bbb1d3c
Merge branch 'develop' into feature/rrfs-lightning
EdwardSnyder-NOAA Mar 30, 2023
29718de
updated lightning root var to epic location
Mar 30, 2023
86a4239
removed unneeded lightning var
Mar 30, 2023
76057ea
removed lightning root
Mar 30, 2023
ff51aea
Merge branch 'develop' into feature/rrfs-lightning
EdwardSnyder-NOAA Apr 4, 2023
ffd1471
Merge branch 'develop' into feature/rrfs-lightning
EdwardSnyder-NOAA Apr 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions devbuild.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,7 @@ CMAKE_SETTINGS="\
-DBUILD_UFS_UTILS=${BUILD_UFS_UTILS}\
-DBUILD_UPP=${BUILD_UPP}\
-DBUILD_GSI=${BUILD_GSI}\
-DBUILD_RRFS_UTILS=${BUILD_RRFS_UTILS}\
-DBUILD_NEXUS=${BUILD_NEXUS}\
-DBUILD_AQM_UTILS=${BUILD_AQM_UTILS}"

Expand Down
96 changes: 96 additions & 0 deletions jobs/JREGIONAL_PROCESS_LIGHTNING
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
#!/bin/bash

#
#-----------------------------------------------------------------------
#
# This J-JOB script runs the NetCDF ligthning observation preprocess
# for the FV3-LAM model
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved
#
#-----------------------------------------------------------------------
#
#
#-----------------------------------------------------------------------
#
# Source the variable definitions file and the bash utility functions.
#
#-----------------------------------------------------------------------
#
. ${USHdir}/source_util_funcs.sh
source_config_for_task "task_process_lightning" ${GLOBAL_VAR_DEFNS_FP}
. ${USHdir}/job_preamble.sh "TRUE"
#
#-----------------------------------------------------------------------
#
# Save current shell options (in a global array). Then set new options
# for this script/function.
#
#-----------------------------------------------------------------------
#
{ save_shell_opts; . $USHdir/preamble.sh; } > /dev/null 2>&1
#
#-----------------------------------------------------------------------
#
# Get the full path to the file in which this script/function is located
# (scrfunc_fp), the name of that file (scrfunc_fn), and the directory in
# which the file is located (scrfunc_dir).
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
#-----------------------------------------------------------------------
#
# Print message indicating entry into script.
#
#-----------------------------------------------------------------------
#
print_info_msg "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"

This is the J-job script for the task that runs a lightning preprocess for
the specified cycle.
========================================================================"
#
#-----------------------------------------------------------------------
#
# Create the working directory under the cycle directory.
#
#-----------------------------------------------------------------------
#
if [ "${CYCLE_TYPE}" == "spinup" ]; then
DATA="${DATA:-${COMIN}/process_lightning_spinup}"
else
DATA="${DATA:-${COMIN}/process_lightning}"
fi
mkdir_vrfy -p ${DATA}
#
#-----------------------------------------------------------------------
#
# Call the ex-script for this J-job and pass to it the necessary varia-
# bles.
#
#-----------------------------------------------------------------------
#
$SCRIPTSdir/exregional_process_lightning.sh || print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
#-----------------------------------------------------------------------
#
# Run job postamble.
#
#-----------------------------------------------------------------------
#
job_postamble
#
#-----------------------------------------------------------------------
#
# Restore the shell options saved at the beginning of this script/func-
# tion.
#
#-----------------------------------------------------------------------
#
{ restore_shell_opts; } > /dev/null 2>&1
207 changes: 207 additions & 0 deletions scripts/exregional_process_lightning.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
#!/bin/bash

#
#-----------------------------------------------------------------------
#
# Source the variable definitions file and the bash utility functions.
#
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_process_lightning" ${GLOBAL_VAR_DEFNS_FP}
#
#-----------------------------------------------------------------------
#
# Save current shell options (in a global array). Then set new options
# for this script/function.
#
#-----------------------------------------------------------------------
#
{ save_shell_opts; . $USHdir/preamble.sh; } > /dev/null 2>&1
#
#-----------------------------------------------------------------------
#
# Get the full path to the file in which this script/function is located
# (scrfunc_fp), the name of that file (scrfunc_fn), and the directory in
# which the file is located (scrfunc_dir).
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( readlink -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
#-----------------------------------------------------------------------
#
# Print message indicating entry into script.
#
#-----------------------------------------------------------------------
#
print_info_msg "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"

This is the ex-script for the task that runs lightning preprocess
with FV3 for the specified cycle.
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved
========================================================================"
#
#-----------------------------------------------------------------------
#
# Extract from CDATE the starting year, month, day, and hour of the
# forecast. These are needed below for various operations.
#
#-----------------------------------------------------------------------
#
START_DATE=$(echo "${CDATE}" | sed 's/\([[:digit:]]\{2\}\)$/ \1/')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have CDATE anymore. The date is now passed with NCO-compliant PDY and cyc variables. For the timestamp variables below, I think that translates to:

YYYYMMDDHH=${PDY}${cyc}
YYYYMMDD=${PDY}
HH=${cyc}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure what the NCO-complaint variables are. But in the develop branch, CDATE is used in a number of ufs-srweather-app/scripts/* and CDATE is defined in the job_preamble.sh script as export CDATE=${PDY}${cyc}.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider CDATE to be non-standard, but tolerated in operations (is pretty widely used in the GFS).

Copy link
Collaborator Author

@EdwardSnyder-NOAA EdwardSnyder-NOAA Mar 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay. That's good to know. I switched the code to be: START_DATE=$(echo "${PDY} ${cyc}")

YYYYMMDDHH=$(date +%Y%m%d%H -d "${START_DATE}")
JJJ=$(date +%j -d "${START_DATE}")

YYYY=${YYYYMMDDHH:0:4}
MM=${YYYYMMDDHH:4:2}
DD=${YYYYMMDDHH:6:2}
HH=${YYYYMMDDHH:8:2}
YYYYMMDD=${YYYYMMDDHH:0:8}

YYJJJHH=$(date +"%y%j%H" -d "${START_DATE}")
PREYYJJJHH=$(date +"%y%j%H" -d "${START_DATE} 1 hours ago")
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved

EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved
#
#-----------------------------------------------------------------------
#
# Get into working directory
#
#-----------------------------------------------------------------------
#
print_info_msg "$VERBOSE" "
Getting into working directory for lightning process ..."

cd_vrfy ${DATA}

fixgriddir=$FIXgsi/${PREDEF_GRID_NAME}

print_info_msg "$VERBOSE" "fixgriddir is $fixgriddir"

#
#-----------------------------------------------------------------------
#
# link or copy background and grid files
#
#-----------------------------------------------------------------------

cp_vrfy ${fixgriddir}/fv3_grid_spec fv3sar_grid_spec.nc

#-----------------------------------------------------------------------
#
# Link to the NLDN data
#
#-----------------------------------------------------------------------
run_lightning=false
filenum=0
LIGHTNING_FILE=${LIGHTNING_ROOT}/vaisala/netcdf
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved
for n in 00 05 ; do
filename=${LIGHTNING_FILE}/${YYJJJHH}${n}0005r
if [ -r ${filename} ]; then
((filenum += 1 ))
ln -sf ${filename} ./NLDN_lightning_${filenum}
else
echo " ${filename} does not exist"
fi
done
for n in 55 50 45 40 35 ; do
filename=${LIGHTNING_FILE}/${PREYYJJJHH}${n}0005r
if [ -r ${filename} ]; then
((filenum += 1 ))
ln -sf ${filename} ./NLDN_lightning_${filenum}
run_lightning=true
else
echo " ${filename} does not exist"
fi
done
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved

echo "found GLD360 files: ${filenum}"

#-----------------------------------------------------------------------
#
# copy bufr table from fix directory
#
#-----------------------------------------------------------------------
BUFR_TABLE=${FIXgsi}/prepobs_prep_RAP.bufrtable

cp_vrfy $BUFR_TABLE prepobs_prep.bufrtable

#-----------------------------------------------------------------------
#
# Build namelist and run executable
#
# analysis_time : process obs used for this analysis date (YYYYMMDDHH)
# NLDN_filenum : number of NLDN lighting observation files
# IfAlaska : logic to decide if to process Alaska lightning obs
# bkversion : grid type (background will be used in the analysis)
# = 0 for ARW (default)
# = 1 for FV3LAM
#-----------------------------------------------------------------------

cat << EOF > namelist.lightning
&setup
analysis_time = ${YYYYMMDDHH},
NLDN_filenum = ${filenum},
grid_type = "${PREDEF_GRID_NAME}",
obs_type = "nldn_nc"
/
EOF

#
#-----------------------------------------------------------------------
#
# Copy the executable to the run directory.
#
#-----------------------------------------------------------------------
#
exec_fn="process_Lightning.exe"
exec_fp="$EXECdir/${exec_fn}"

if [ ! -f "${exec_fp}" ]; then
print_err_msg_exit "\
The executable specified in exec_fp does not exist:
exec_fp = \"${exec_fp}\"
Build lightning process and rerun."
fi
#
#
#-----------------------------------------------------------------------
#
# Run the process
#
#-----------------------------------------------------------------------
#

if [[ "$run_lightning" == true ]]; then
PREP_STEP
eval $RUN_CMD_UTILS --account=${ACCOUNT} --nodes=1-1 --tasks-per-node=1 --time 00:30:00 ${exec_fp} ${REDIRECT_OUT_ERR} || print_err_msg "\
Call to executable to run lightning (nc) process returned with nonzero exit code."
EdwardSnyder-NOAA marked this conversation as resolved.
Show resolved Hide resolved
POST_STEP
fi
#
#-----------------------------------------------------------------------
#
# Print message indicating successful completion of script.
#
#-----------------------------------------------------------------------
#
print_info_msg "
========================================================================
LIGHTNING PROCESS completed successfully!!!

Exiting script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
========================================================================"
#
#-----------------------------------------------------------------------
#
# Restore the shell options saved at the beginning of this script/func-
# tion.
#
#-----------------------------------------------------------------------
#
{ restore_shell_opts; } > /dev/null 2>&1
1 change: 1 addition & 0 deletions ush/valid_param_vals.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,3 +105,4 @@ valid_vals_DO_AQM_CHEM_LBCS: [True, False]
valid_vals_DO_AQM_GEFS_LBCS: [True, False]
valid_vals_DO_AQM_SAVE_AIRNOW_HIST: [True, False]
valid_vals_COLDSTART: [True, False]
valid_vals_DO_NLDN_LGHT: [True, False]