Add global_cycle job and script to global-workflow

[RussTreadon-NOAA]

This issue is opened to document the move of global_cycle and gaussian_sfcanl j-jobs and scripts from NOAA-EMC/GSI into NOAA-EMC/global-workflow. This refactoring is in support of the goals stated in the JEDI Prototype 3DVar Cycling Capability project.

[RussTreadon-NOAA]

Create feature/add_sfcanl in RussTreadon-NOAA/global-workflow. feature/add_sfcanl is a copy of the NOAA-EMC/global-workflow develop at dd03ed0.

[RussTreadon-NOAA]

As discussed in NOAA-EMC/GSI issue #266, reduce the scope of this issue to only move global_cycle into its own job (see specific comment here).

50b53b5 commits changes required to run global_cycle as its own job.

These changes were tested on WCOSS_D using warm start parallel using operational ICs from 2021121306. The control used the current heads of the global-workflow develop (dd03ed0) and NOAA-EMC/GSI master (a62dec6). The test used NOAA-EMC/GSI forked branch rm_sfcanl at f0161c8 and global-workflow forked branch add_sfcanl at 50b53b5.

The control and test both ran the gfs and gdas prep, anal, and analcalc jobs for 2021121312. The test also ran the new job, sfcanl. sfcanl executes global_cycle. The test reproduced the control sfcanl tiles in the gfs and gdas RESTART directories. The test also reproduced the control gfs and gdas sfcanl.nc files. The test was not cycled via a cron. Each job was manually submitted. A cycled test driven by a cron needs to be run to test job dependencies.

[RussTreadon-NOAA]

Cycled testing has been completed on Mars. A control, cntrl, and test, sfcanl were set up as follows

• cntrl
  • global-workflow develop at dd03ed0
  • gsi master at eed3d40
• sfcanl
  • forked global-workflow feature/add_sfcanl at 512ae81
  • forked gsi feature/rm_sfcanl at a75d6ed

Both parallels were warm stared with operational ICs for 2021121306. Both parallels began with the 2021121312 gdas prep and ran through the end of the 2021121400 gfs and gdas cycles. The gfs was only run for 2021121400. The files needed to run the 00Z gldas were taken from operations.

A check of 12Z, 18Z, and 00Z files show that sfcanl generated identical analysis and forecasts with respect to cntrl. The change in job dependencies introduced by the addition of the gdas and gfs sfcanl jobs were correctly accounted for in sfcanl.

The combination of changes in NOAA-EMC/GSI issue #266 and global-workflow issue #527 are working as intended based on single cycle and cycled testing.

One note: Neither the cntrl or sfcanl ran with WAVES on. These were atmosphere only cycled parallels. This mismatch caused the 2021121312 gdas and enkf forecasts to fail with checksum errors. For example,

[288] FATAL from PE   288: The checksum in the file:INPUT/fv_core.res.tile4.nc and variable:u does not match the checksum calculated from the data. file:82D373ED39EFA0E3 from data:B6C15CF1A0000000

Walter pointed me to pre-processing to complete prior to using warm start operational ICs. Once this procedure was applied to the fv_core.res.tile*.nc, fv_tracer.res.tile*.nc, and fv_srf_wnd.res.tile*.nc tiles the 2021121312 forecasts ran to completion.

[RussTreadon-NOAA]

As explained in NOAA-EMC/GSI issue #266, neither #266 nor this issue, #527, may enter their parent repos (master and develop, respectively) until a GFS DA v16.x release branch is created in NOAA-EMC/GSI.

The forked workflow branch, add_sfcanl, will be kept in sync with develop updates until PRs for issues #266 and #527 are created, reviewed, and closed.

[aerorahul]

closed with #905