Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional fixes for downstream jobs #3187

Open
wants to merge 31 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
5974b96
Only archive gempak if it was run
DavidHuber-NOAA Dec 20, 2024
2649a86
Remove literal quotes
DavidHuber-NOAA Dec 20, 2024
a038cb9
Shellcheck stuff
DavidHuber-NOAA Dec 20, 2024
1180627
Remove duplicate marinebmat entry
DavidHuber-NOAA Dec 20, 2024
1efe711
Add DO_GEMPAK to archive environment
DavidHuber-NOAA Dec 20, 2024
4b7d39e
Merge remote-tracking branch 'emc/develop' into fix/awips
DavidHuber-NOAA Dec 20, 2024
4df2811
Fix fbwinds resource assignments
DavidHuber-NOAA Dec 24, 2024
939c238
Source preamble in gfs_bufr.sh
DavidHuber-NOAA Dec 24, 2024
66440fd
Merge remote-tracking branch 'emc/develop' into fix/awips
DavidHuber-NOAA Dec 30, 2024
52a1225
Remove duplicate preamble sourcing
DavidHuber-NOAA Dec 30, 2024
28e7775
Enable AWIPS for extended CI test
DavidHuber-NOAA Dec 30, 2024
0796a09
Source config.fbwnd if running AWIPS in forecast only mode
DavidHuber-NOAA Jan 2, 2025
6c3fb14
Merge branch 'develop' into fix/awips
aerorahul Jan 3, 2025
06bc9dd
Merge branch 'develop' into fix/awips
DavidHuber-NOAA Jan 6, 2025
66a0f86
Merge remote-tracking branch 'emc/develop' into fix/awips
DavidHuber-NOAA Jan 7, 2025
6d746b6
Merge remote-tracking branch 'emc/develop' into HEAD
DavidHuber-NOAA Jan 7, 2025
a1831e5
Apply multiple fixes for the gempakncdcupapgif job
DavidHuber-NOAA Jan 8, 2025
8187b7a
Address shellcheck issues
DavidHuber-NOAA Jan 8, 2025
1309cc0
Add missing braces
DavidHuber-NOAA Jan 8, 2025
1488020
Merge remote-tracking branch 'emc/develop' into fix/awips
DavidHuber-NOAA Jan 8, 2025
ce3924a
Replace COM with COMIN/OUT for the wavegempak jjob
DavidHuber-NOAA Jan 10, 2025
cdea635
Fix multiple issues with waveawipsgridded job
DavidHuber-NOAA Jan 10, 2025
ed10ae1
Replace COM with COMIN/OUT for wavegempak script; add glo_200 grid ru…
DavidHuber-NOAA Jan 10, 2025
ec02b2e
Group gempak/goes tasks
DavidHuber-NOAA Jan 10, 2025
4b1551f
Remove obs-dependent gempak jobs from forecast-only
DavidHuber-NOAA Jan 10, 2025
3444223
Fix cleanup dependencies for gempak jobs
DavidHuber-NOAA Jan 10, 2025
82f8705
Merge remote-tracking branch 'emc/develop' into fix/awips
DavidHuber-NOAA Jan 10, 2025
3bbcc41
Remove shellcheck directive
DavidHuber-NOAA Jan 10, 2025
5789de3
Update gfs_utils hash
DavidHuber-NOAA Jan 10, 2025
5265d98
Add C48_S2SW_extended test case
DavidHuber-NOAA Jan 10, 2025
8905bfa
Clarify COM directory creation
DavidHuber-NOAA Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/cases/yamls/gfs_extended_ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ base:
DO_GOES: "YES"
DO_BUFRSND: "YES"
DO_GEMPAK: "YES"
DO_AWIPS: "NO"
DO_AWIPS: "YES"
DO_NPOESS: "YES"
DO_GENESIS_FSU: "NO"
FCST_BREAKPOINTS: 192
Expand Down
5 changes: 0 additions & 5 deletions env/HERA.env
Original file line number Diff line number Diff line change
Expand Up @@ -133,11 +133,6 @@ elif [[ "${step}" = "marinebmat" ]]; then
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
export APRUN_MARINEBMAT="${APRUN_default}"

elif [[ "${step}" = "marinebmat" ]]; then

export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
export APRUN_MARINEBMAT="${APRUN_default}"

elif [[ "${step}" = "marineanlvar" ]]; then

export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
Expand Down
2 changes: 2 additions & 0 deletions parm/archive/gfs_downstream.yaml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ gfs_downstream:
name: "GFS_DOWNSTREAM"
target: "{{ ATARDIR }}/{{ cycle_YMDH }}/gfs_downstream.tar"
required:
{% if DO_GEMPAK %}
- "{{ COMIN_ATMOS_GEMPAK | relpath(ROTDIR) }}/gfs_{{ cycle_YMDH }}.sfc"
- "{{ COMIN_ATMOS_GEMPAK | relpath(ROTDIR) }}/gfs_{{ cycle_YMDH }}.snd"
{% endif %}
{% for i in range(1, NUM_SND_COLLECTIVES) %}
- "{{ COMIN_ATMOS_BUFR | relpath(ROTDIR) }}/gfs_collective{{ i }}.fil"
{% endfor %}
KateFriedman-NOAA marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
1 change: 1 addition & 0 deletions parm/config/gfs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -1288,6 +1288,7 @@ case ${step} in
ntasks=1
threads_per_task=1
memory="4GB"
tasks_per_node=1
;;

"mos_stn_prep")
Expand Down
2 changes: 1 addition & 1 deletion scripts/exgfs_atmos_postsnd.sh
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ for fhr in "${hour_list[@]}"; do
echo "File ${filename} is required but not found."
err_exit "FATAL ERROR: logf${fhr} not found."
else
echo "${runscript} \"${fhr}\" \"${fhr_p}\" \"${FINT}\" \"${F00FLAG}\" \"${DATA}/${fhr}\"" >> "${DATA}/poescript_bufr"
echo "${runscript} ${fhr} ${fhr_p} ${FINT} ${F00FLAG} ${DATA}/${fhr}" >> "${DATA}/poescript_bufr"
fi
done

Expand Down
2 changes: 1 addition & 1 deletion scripts/exglobal_archive.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ def main():
'NMEM_ENS', 'DO_JEDIATMVAR', 'DO_VRFY_OCEANDA', 'FHMAX_FITS', 'waveGRD',
'IAUFHRS', 'DO_FIT2OBS', 'NET', 'FHOUT_HF_GFS', 'FHMAX_HF_GFS', 'REPLAY_ICS',
'OFFSET_START_HOUR', 'ARCH_EXPDIR', 'EXPDIR', 'ARCH_EXPDIR_FREQ', 'ARCH_HASHES',
'ARCH_DIFFS', 'SDATE', 'EDATE', 'HOMEgfs']
'ARCH_DIFFS', 'SDATE', 'EDATE', 'HOMEgfs', 'DO_GEMPAK']

archive_dict = AttrDict()
for key in keys:
Expand Down
15 changes: 8 additions & 7 deletions ush/gfs_bufr.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,17 @@
# 2024-08-08 Bo Cui: Update to handle one forecast at a time
# echo "History: February 2003 - First implementation of this utility script"
#
fhr="$1"
fhr_p="$2"
FINT="$3"
F00FLAG="$4"
workdir="$5"

cd "${workdir}" || exit 2

source "${USHgfs}/preamble.sh"

fhr="${1}"
fhr_p="${2}"
FINT="${3}"
F00FLAG="${4}"
workdir="${5}"

cd "${workdir}" || exit 2

if [[ "${F00FLAG}" == "YES" ]]; then
f00flag=".true."
else
Expand Down
2 changes: 1 addition & 1 deletion workflow/applications/gfs_cycled.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ def _get_app_configs(self, run):
configs += ['postsnd']

if options['do_awips']:
configs += ['awips']
configs += ['awips', 'fbwind']
aerorahul marked this conversation as resolved.
Show resolved Hide resolved

if options['do_wave']:
configs += ['waveinit', 'waveprep', 'wavepostsbs', 'wavepostpnt']
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/hera.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
MAKE_ACFTBUFR: 'NO'
DO_AWIPS: 'NO'
SUPPORTED_RESOLUTIONS: ['C1152', 'C768', 'C384', 'C192', 'C96', 'C48']
COMINecmwf: /scratch1/NCEPDEV/global/glopara/data/external_gempak/ecmwf
COMINnam: /scratch1/NCEPDEV/global/glopara/data/external_gempak/nam
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/hercules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ MAKE_NSSTBUFR: 'NO'
MAKE_ACFTBUFR: 'NO'
DO_TRACKER: 'NO'
DO_GENESIS: 'NO'
DO_AWIPS: 'NO'
SUPPORTED_RESOLUTIONS: ['C1152', 'C768', 'C384', 'C192', 'C96', 'C48']
COMINecmwf: /work/noaa/global/glopara/data/external_gempak/ecmwf
COMINnam: /work/noaa/global/glopara/data/external_gempak/nam
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/orion.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ MAKE_NSSTBUFR: 'NO'
MAKE_ACFTBUFR: 'NO'
DO_TRACKER: 'NO'
DO_GENESIS: 'NO'
DO_AWIPS: 'NO'
SUPPORTED_RESOLUTIONS: ['C1152', 'C768', 'C384', 'C192', 'C96', 'C48']
COMINecmwf: /work/noaa/global/glopara/data/external_gempak/ecmwf
COMINnam: /work/noaa/global/glopara/data/external_gempak/nam
Expand Down
2 changes: 1 addition & 1 deletion workflow/rocoto/gfs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -1410,7 +1410,7 @@ def fbwind(self):
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps, dep_condition='and')

resources = self.get_resource('awips')
resources = self.get_resource('fbwind')
# TODO: It would be better to use task dependencies on the
# individual post jobs rather than data dependencies to avoid
# prematurely starting with partial files. Unfortunately, the
Expand Down