Pick up module version changes from NCO based on WAFSv7.0.0

[YaliMao-NOAA]

Pick up module version updates from NCO based on WAFSv7.0.0, to address UPP issue #1089.
Outputs were compared and they are bit-wise identical.

[ChristopherHill-NOAA]

Modifications were made to available regression testing scripts (previously developed by @WenMeng-NOAA) in an attempt to conform to the code structure the WAFS-based version of UPP featured in this pull request. The UPP code was compiled in local disk space on WCOSS2 (Dogwood), using sorc/build_ncep_post.sh.

Initial UPP regression tests were performed using F006 data files that resulted from the 20241114 00 UTC run of the operational GFS. Most of the files generated from the RTs were indicated as being not bit-identical. In particular, the wafs files were found to be significantly different in size:

original files (size in bytes):

3074689370   gfs.t00z.master.grb2f006.WCOSS2
 304624574   gfs.t00z.sfluxgrbf006.grib2.WCOSS2
  15121679   gfs.t00z.special.grb2f006.WCOSS2
 968468322   gfs.t00z.wafs.grb2f006.WCOSS2
 452854338   gfs.t00z.wafs_icao.grb2f006.WCOSS2

from RTs (size in bytes):

3080995479   gfs.t00z.master.grb2f006
 304951495   gfs.t00z.sfluxgrbf006.grib2
  15121679   gfs.t00z.special.grb2f006
 804254188   gfs.t00z.wafs.grb2f006
 271237086   gfs.t00z.wafs_icao.grb2f006

It is recognized that WAFS v7 operates independently of GFS, and that the UPP output from WAFS takes the form of wafs.t00z.master.f006.grib2. Such output files were not generated in these RTs.

Regression testing with specific focus to the WAFS v7 system - and exclusive of the GFS - is likely needed to perform accurate tests of the UPP submodule. @WenMeng-NOAA may have already developed RT scripts that are appropriate for this pull request.

@HuiyaChuang-NOAA is copied here for awareness, and the opportunity to provide feedback.

[YaliMao-NOAA]

@ChristopherHill-NOAA Sorry I didn't introduce the background that this WAFS.v7 version is based on operational release/gfs_v16. So the regression test should be based on the operational products, not based on the canned products from UPP develop branch.

[ChristopherHill-NOAA]

It was noted that the only changes in this PR were confined to the file modulefiles/post/post_wcoss2.lua, and it is expected that such changes will result in nearly bit-identical output from the UPP.

[ChristopherHill-NOAA]

@ChristopherHill-NOAA Sorry I didn't introduce the background that this WAFS.v7 version is based on operational release/gfs_v16. So the regression test should be based on the operational products, not based on the canned products from UPP develop branch.

@YaliMao-NOAA - As noted earlier, the F006 data files 2024111400 GFS were used in my RTs. They are indicated as *.WCOSS2 files in my initial posting. Can you provide log files from your runs of WAFS/UPP that resulted in bit-identical UPP output?

[YaliMao-NOAA]

@ChristopherHill-NOAA WAFS UPP is not generating GFS master file like gfs.t00z.master.grb2f006.
WAFSv7.0 UPP is generating WAFS master file,
/lfs/h2/emc/ptmp/yali.mao/wafsxPDY/prod/com/wafs/v7.0/wafs.20241114/00/upp on Dogwood.
which matches the operational GFSv16.3 /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20241114/00/atmos/gfs.t00z.wafs.grb2f006

However the WAFSv7.0 master files have more fields than the GFSv16.3 wafs files. So I created a script to extract the fields available in GFSv16.3 wafs files from WAFSv7.0 master files, then compare the extracted WAFSv7.0 master files to GFSv16.3 wafs files which are supposed to bit-wise identical.
The script is /lfs/h2/emc/vpppg/noscrub/yali.mao/git/save/scripts/compare2folders.sh

[YaliMao-NOAA]

@ChristopherHill-NOAA The other simpler way to do regression tests on this PR is to check that the outputs are bit-wise identical before and after.

[ChristopherHill-NOAA]

After the UPP code is compiled and run in local disk space on WCOSS2, the resulting gfs.t{cyc}z.wafs.grib2.f006 files are being compared against those files available within the directory:
/lfs/h1/ops/{prod,para}/com/gfs/v16.3/gfs.{yyyymmdd}/{cyc}/atmos

UPP output specific to WAF v7 are being found at:
/lfs/h1/ops/para/com/wafs/v7.0/wafs.{yyyymmdd}/upp/wafs.t{cyc}z.master.f006.grib2

Versions of run_JGLOBAL_NCEPPOST and submit_run_JGLOBAL_NCEPPOST.sh - modified for use on WCOSS2 - were found in the disk space of @WenMeng-NOAA and used to perform a new regression test for cycle 20241120 00 UTC. Differences were again found between the prod, para output and the output generated by the RT in local disk space:

20241120 00 prod:    979784387 bytes    gfs.t00z.wafs.grb2f006
20241120 00 para:    977575398 bytes    gfs.t00z.wafs.grb2f006
from the RT:         815168592 bytes    gfs.t00z.wafs.grb2f006

@YaliMao-NOAA I suspect I may have compiled your UPP branch wafs_v7_moduleupdate in a manner inconsistent with prod and para, as I do not believe that the references to new library versions in modulefiles/post/post_wcoss2.lua would cause the observed file size differences. Are you able to share a log file, or your version of build_ncep_post.sh, that shows the parameter settings used for UPP compilation? Was your branch of UPP compiled through the WAFS directory structure? Might there be parameter settings in WAFS v7 that control UPP compilation?

[ChristopherHill-NOAA]

From WAFS v7 ...
20241120 00 para: 987282287 bytes wafs.t00z.master.f006.grib2

[YaliMao-NOAA]

@ChristopherHill-NOAA
I have two log files for WAFS.v7, one is NCO para, the other is mine:
NCO: /lfs/h1/ops/para/output/20241120/wafs_upp_f006_00.o162685369
Mine: /lfs/h2/emc/stmp/yali.mao/wafs/output/wafs_upp_f006_00.o206915164

NCO para output:
/lfs/h1/ops/para/com/wafs/v7.0/wafs.20241120/00/upp/wafs.t00z.master.f006.grib2
My output:
/lfs/h2/emc/ptmp/yali.mao/wafs/prod/com/wafs/v7.0/wafs.20241120/00/upp/wafs.t00z.master.f006.grib2
Both are 987282287 bytes and bit-wise identical.

They are different from /lfs/h1/ops/{prod,para}/com/gfs/v16.3/gfs.{yyyymmdd}/{cyc}/atmos, for the reason that I explained earlier:

However the WAFSv7.0 master files have more fields than the GFSv16.3 wafs files. So I created a script to extract the fields available in GFSv16.3 wafs files from WAFSv7.0 master files, then compare the extracted WAFSv7.0 master files to GFSv16.3 wafs files which are supposed to bit-wise identical.
The script is /lfs/h2/emc/vpppg/noscrub/yali.mao/git/save/scripts/compare2folders.sh

[YaliMao-NOAA]

@ChristopherHill-NOAA I am looking at the difference of gfs.t00z.wafs.grb2f006 between GFS prod and para

[YaliMao-NOAA]

@ChristopherHill-NOAA I am looking at the difference of gfs.t00z.wafs.grb2f006 between GFS prod and para

@ChristopherHill-NOAA The operational GFS is v16.3.19 while parallel GFS is v16.3.20 which has observation data update. That's why GFS para and prod gfs.t00z.wafs.grb2f006 are different.
NOAA-EMC/global-workflow@gfs.v16.3.19...EMC-v16.3.20

Please compare WAFSv7 to the operational GFS output

[ChristopherHill-NOAA]

From a review of the log file /lfs/h2/emc/stmp/yali.mao/wafs/output/wafs_upp_f006_00.o206915164, it appears that the version of UPP featured in this pull request is compiled and run through the WAF v7 directory structure. However, the code branches available within UPP submodule of WAFS and the UPP repository differ.

The WAFS repository was cloned into local disk space on WCOSS2, and the command git checkout release/wafs_v7 was applied. Following the "checkout" instructions provided within the WAFS Github page, the file sorc/checkout_upp.sh was executed from the WAFS top-level directory. After entering into the filled wafs_upp.fd directory, execution of git branch -a does not show branch wafs_v7_moduleupdate to be among the available branches.

@YaliMao-NOAA Is it possible to clone - or otherwise retrieve - the content of branch wafs_v7_moduleupdate within the cloned WAFS repository using sorc/checkout_upp.sh? Are you able to generate your WAFS-UPP output using the stand-alone version of UPP?

I apologize for the fact that this PR effort has become an extended learning exercise on my part, and may have caused unnecessary delay to the completion of the PR.

[YaliMao-NOAA]

@ChristopherHill-NOAA You are right. That's the reason why I need a tag of UPP for WAFS from you, then I will update WAFSv7 branch with the new UPP tag. Then you can see the change updated when cloning and checking out release/wafs_v7

[WenMeng-NOAA]

@YaliMao-NOAA I have run the UPP standalone test with model input data from GFS V16. My test results are at
/u/wen.meng/ptmp/com-after/gfs.20241125/00 on Dogwood. Please validate my test.

[YaliMao-NOAA]

@WenMeng-NOAA May I know your logfile? The outputs are different from the operational which is not expected to be.
Without the module version updates, are you able to duplicate the operational outputs?

[WenMeng-NOAA]

@WenMeng-NOAA May I know your logfile? The outputs are different from the operational which is not expected to be. Without the module version updates, are you able to duplicate the operational outputs?

@YaliMao-NOAA I have done two UPP standalone tests on Dogwood:

1. test 1: model history files from ops. gfs, UPP version from the UPP release/wafs_v7 branch
   runtime directory: /u/wen.meng/ptmp/test/fv3gfs.20241125_rundir-before
   results: /u/wen.meng/ptmp/com-before/gfs.20241125/00

2. test 2: model history file from ops. gfs, UPP version from your branch
   runtime directory: /u/wen.meng/ptmp/test/fv3gfs.20241125_rundir-after
   results: /u/wen.meng/ptmp/com-after/gfs.20241125/00

The wafs related data files generated from my tests are bite-wise indentical.

[YaliMao-NOAA]

@WenMeng-NOAA Thank you for the test runs. Thank you for pointing out the reason for your test output differences from the operational is because the operational is inline post while the test run is offline standalone post.
Since the outputs of 'before' and 'after' the module versions are bitwise identical, this PR will not change the product outputs, as expected. The PR is good to go.

Please generate a new UPP tag for WAFS, upp_wafs_v7.0.1, after merging this PR.
Thank you!

[WenMeng-NOAA]

This PR is ready for merging.

[WenMeng-NOAA]

@YaliMao-NOAA The tag "upp_wafs_v7.0.1" was created.

[YaliMao-NOAA]

@WenMeng-NOAA Thank you!