Populate g-w fix/gdas with GSI static-B files for UFS DA applications

[RussTreadon-NOAA]

Description
GMAO added the ability to use the GSI background error covariance file (staticb) in JEDI applications. GDASApp issue #166 adds the use of the GSI staticb to the GDASApp.

Requirements
In order to use GSI staticb in fv3jedi_var.x from g-w we need to populate the g-w gdas fix directory with the gsibec fix directory. Below are the source and target directories and action to take on each machine.

Hera

• SRCDIR=/scratch1/NCEPDEV/da/Cory.R.Martin/GDASApp/fix/gsibec
• TARGET=/scratch1/NCEPDEV/global/glopara/fix/gdas
• action: cp -rp $SRCDIR $TARGET/

Orion

• SRCDIR=/work2/noaa/da/cmartin/GDASApp/fix/gsibec
• TARGET=/work/noaa/global/glopara/fix/gdas
• action: cp -rp $SRCDIR $TARGET/

Acceptance Criteria (Definition of Done)
This issue can be closed when the gsibec fix directory is copied to the g-w gdas fix directory as described above.

Dependencies
None

[KateFriedman-NOAA]

Pulled into the fix set on Hera (new gsibec/20221031 subfolder):

[glopara@hfe07 gdas]$ pwd
/scratch1/NCEPDEV/global/glopara/fix/gdas
[glopara@hfe07 gdas]$ ll gsibec/20221031/
total 12
drwxr-sr-x 2 glopara global 4096 Oct 28 15:21 C192
drwxr-sr-x 2 glopara global 4096 Oct 28 15:21 C384
drwxr-sr-x 2 glopara global 4096 Oct 28 15:21 C96
[glopara@hfe07 gdas]$ ll gsibec/20221031/*                                                                                                                       
gsibec/20221031/C192:
total 15832
-rw-r--r-- 1 glopara global     3593 Oct 28 15:11 gfs_gsi_global.nml
-rw-r--r-- 1 glopara global 16207256 Oct 28 15:03 gsi-coeffs-gfs-global-l127x384y192.nc4

gsibec/20221031/C384:
total 32976
-rw-r--r-- 1 glopara global     3595 Oct 28 15:08 gfs_gsi_global.nml
-rw-r--r-- 1 glopara global 33761200 Oct 28 15:03 gsi-coeffs-gfs-global-l127x768y386.nc4

gsibec/20221031/C96:
total 7780
-rw-r--r-- 1 glopara global    3592 Oct 28 14:15 gfs_gsi_global.nml
-rw-r--r-- 1 glopara global 7960088 Oct 27 17:13 gsi-coeffs-gfs-global-l127x192y96.nc4

Will rsync to the other platforms next.

Will need to add this new subfolder into the workflow layer now (add gdas_gsibec_ver=20221031 into fix.ver). Are you planning to make other workflow-related updates @RussTreadon-NOAA ? Do any scripts need gdas_gsibec_ver?

[RussTreadon-NOAA]

Good point, @KateFriedman-NOAA . You are right. Now that g-w maintains jobs and scripts, we need to update scripts/exgdas_global_atmos_analysis_run.sh. It is currently hard-wires static-b to bump

# link fix files to $DATA
# static B
CASE_BERROR=${CASE_BERROR:-${CASE_ANL:-$CASE}}
$NLN $FV3JEDI_FIX/bump/$CASE_BERROR/ $DATA/berror

We added a new variable, staticb_type, to the GDASApp to toggle between various static-b sources. We need to add the same flexibility to g-w. Let's try the following:

1. add STATICB_TYPE to g-w config.atmanal

export ATMVARYAML=$HOMEgfs/sorc/gdas.cd/parm/atm/variational/3dvar_dripcg.yaml
export STATICB_TYPE=bump
export BERROR_YAML=$HOMEgfs/sorc/gdas.cd/parm/atm/$STATICB_TYPE/staticb_identity.yaml
export FV3JEDI_FIX=$HOMEgfs/fix/gdas

Valid values for STATICB_TYPE are bump, gsibec, and identity. The identity model is internal to UFS DA applications. There is no berror fix directory for identity.

2. Change exgdas_global_atmos_analysis_run.sh to

# link fix files to $DATA
# static B
CASE_BERROR=${CASE_BERROR:-${CASE_ANL:-$CASE}}
$NLN $FV3JEDI_FIX/$STATICB_TYPE/$CASE_BERROR/ $DATA/berror

Maintaining component jobs and scripts separately from component repos has pros and cons.

[RussTreadon-NOAA]

Create g-w branch feature/staticb to which the above changes will be committed.

[KateFriedman-NOAA]

Also need to incorporate the relevant fix.ver variables so you get the correct timestamp subfolder in those paths when linking to the bump/crtm/fv3jedi/gsibec.

In your example you appear to have replaced the hardcoded bump with berror but berror isn't a fix/gdas subfolder. You mention berror, gsibec, and identity...where do berror and identity come from?

An example NLN line where STATICB_TYPE=gsibec:

# link fix files to $DATA
# static B
CASE_BERROR=${CASE_BERROR:-${CASE_ANL:-$CASE}}
STATICB_TYPE=gsibec
STATICB_TYPE_VER="gdas_${STATICB_TYPE}_ver"
$NLN $FV3JEDI_FIX/$STATICB_TYPE/${!STATICB_TYPE_VER}/$CASE_BERROR/ $DATA/berror

^ this is just an example for the files coming from staged fix

Will also need to add gsibec to link_workflow.sh: https://github.com/NOAA-EMC/global-workflow/blob/develop/sorc/link_workflow.sh#L182

Would you like my assistance making fix version updates in feature/staticb?

[RussTreadon-NOAA]

Thank, @KateFriedman-NOAA , for offering to make the fix_ver changes to feature/staticb. I just did so. I tested the changes in a working copy. They behaved as I think they should. A check that I got it right would be appreciated. I will open a PR for these changes.

[KateFriedman-NOAA]

Okie dokie, make me a reviewer on the PR and I'll check the updates. Thanks!

[RussTreadon-NOAA]

Also need to incorporate the relevant fix.ver variables so you get the correct timestamp subfolder in those paths when linking to the bump/crtm/fv3jedi/gsibec.

In your example you appear to have replaced the hardcoded bump with berror but berror isn't a fix/gdas subfolder. You mention berror, gsibec, and identity...where do berror and identity come from?

An example NLN line where STATICB_TYPE=gsibec:

# link fix files to $DATA
# static B
CASE_BERROR=${CASE_BERROR:-${CASE_ANL:-$CASE}}
STATICB_TYPE=gsibec
STATICB_TYPE_VER="gdas_${STATICB_TYPE}_ver"
$NLN $FV3JEDI_FIX/$STATICB_TYPE/${!STATICB_TYPE_VER}/$CASE_BERROR/ $DATA/berror

^ this is just an example for the files coming from staged fix

Will also need to add gsibec to link_workflow.sh: https://github.com/NOAA-EMC/global-workflow/blob/develop/sorc/link_workflow.sh#L182

Would you like my assistance making fix version updates in feature/staticb?

Thank you, @KateFriedman-NOAA , for catching my error. I updated the comment.

[RussTreadon-NOAA]

Hmm, this is messy. I need to take a step back and rethink how to cleanly and consistently toggle between various background error configurations for UFS DA in both GDASApp and g-w. What I have now works well in GDASApp. It's clumsy in g-w.

[RussTreadon-NOAA]

Test feature/staticb in Hera UFS-DA parallel for STATICB_TYPE options identity, bump, and gsibec.

GDASApp issue #166 points out that fv3jedi_var.x fails when run with bump staticb files. This noted, the atmanalrun run directory was correctly populated with bump staticb files when STATICB_TYPE=bump in config.atmanal. The reason(s) for the fv3jedi_var.x faiilure with the existing set of bump files and yamls will be will be investigated in a future GDASApp issue.

UFS DA fv3jedi_var.x runs to completion with STATICB_TYPE set to identity or gsibec. The gsibec test revealed the need for additional logic in exgdas_global_atmos_analysis_run.sh to handle the case of the fv3jedi_var.x analysis not being performed at the ensemble resolution.

The identity staticb is less computationally intensive. As such STATICB_TYPE in config.atmanl is changed to identity.