Bring GFS.v16 features back to develop branch #274

junwang-noaa · 2020-11-12T18:07:29Z

Description

This PR merges the GFS.v16 dycore changes back to the EMC dev/emc branch. Also nceppost is changed to upp version in GFSv16.
Due to the upp lib and post control file changes, the results are expected to change for post RT test, no change for other test.

Issue(s) addressed

#251 : Bring GFS.v16 features back to develop branch

Testing

RT tests have been run on supported platforms; hera, orion, dell and cray.
The post RT test has results change while other RT tests have not.

Dependencies

Associate PRs:
dycore PR #44
fv3atm PR #196
ufs-weather-model PR #274

… cray

WenMeng-NOAA

The UPP part looks good to me.

WenMeng-NOAA · 2020-11-12T19:57:56Z

@junwang-noaa I saw some updates of paths of hpc-stack in modulefiles. I assume with this PR merged, the develop will be used for the upcoming developments e.g. GFS V17?

junwang-noaa · 2020-11-12T20:42:32Z

Wen, Yes, the new hpc-stack has the updated upp lib. After this PR is merged, the develop branch will be used for GFS.v17 development. Jun

…

On Thu, Nov 12, 2020 at 2:58 PM WenMeng-NOAA ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> I saw some updates of paths of hpc-stack in modulefiles. I assume with this PR merged, the develop will be used for the upcoming developments e.g. GFS V17? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TP2XCGDF2DQ4FQ3UY3SPQ45JANCNFSM4TTTQ4JA> .

WenMeng-NOAA · 2020-11-12T22:09:37Z

Jun, Thanks for clarifying. I will update the UPP develop having consistent hpc-stack configuration with the ufs-weather-model. Wen

…

On Thu, Nov 12, 2020 at 3:42 PM Jun Wang ***@***.***> wrote: Wen, Yes, the new hpc-stack has the updated upp lib. After this PR is merged, the develop branch will be used for GFS.v17 development. Jun On Thu, Nov 12, 2020 at 2:58 PM WenMeng-NOAA ***@***.***> wrote: > @junwang-noaa <https://github.com/junwang-noaa> I saw some updates of > paths of hpc-stack in modulefiles. I assume with this PR merged, the > develop will be used for the upcoming developments e.g. GFS V17? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #274 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AI7D6TP2XCGDF2DQ4FQ3UY3SPQ45JANCNFSM4TTTQ4JA > > . > — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQGNERULKYLGC23SFQYVILSPRCENANCNFSM4TTTQ4JA> .

-- Wen Meng IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct., Room 2070 College Park, MD 20740 Wen.Meng@noaa.gov 301-683-3779

DusanJovic-NOAA · 2020-11-13T14:47:48Z

modulefiles/hera.intel/fv3

@@ -15,13 +15,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc
 setenv CMAKE_Fortran_COMPILER mpiifort
 setenv CMAKE_Platform hera.intel

-module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
+module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack


Please use the same naming convention for hpc-stack installation (lib/hpc-stack/vX.Y.Z). hpc-stack repo must have a tag pointing to that exact version number.

junwang-noaa · 2020-11-13T15:03:37Z

This is the test version I can get so far, we need to get the code changes in as other developer groups are waiting for some time. We can update to a tag version when it is ready.

…

On Fri, Nov 13, 2020 at 9:48 AM Dusan Jovic ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In modulefiles/hera.intel/fv3 <#274 (comment)> : > @@ -15,13 +15,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc setenv CMAKE_Fortran_COMPILER mpiifort setenv CMAKE_Platform hera.intel -module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack +module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack Please use the same naming convention for hpc-stack installation (lib/hpc-stack/vX.Y.Z). hpc-stack repo must have a tag pointing to that exact version number. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TM5PXJ2RXGHSI35P73SPVBKVANCNFSM4TTTQ4JA> .

gfsv16_20201015: modulefiles and regression test logs for cheyenne.{intel,gnu} and hera.{intel,gnu}

junwang-noaa · 2020-11-17T00:45:33Z

Thanks, Dom. I will copy the baseline to hera. My orion job has not finished yet.

…

On Mon, Nov 16, 2020 at 5:57 PM Dom Heinzeller ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> I added the hera regression test logs to my PR junwang-noaa#5 <junwang-noaa#5> for your branch - please merge at your convenience. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TKVUS3RQNUU6SPGIYTSQGU7FANCNFSM4TTTQ4JA> .

climbfuji · 2020-11-17T03:08:25Z

Thanks, Dom. I will copy the baseline to hera. My orion job has not finished yet.
…

No problem. I already started copying the baseline directories on hera, but it's very slow again:

rsync -av /scratch1/BMC/gmtb/Dom.Heinzeller/FV3_RT/TMP_PR274_BASELINE_GNU/ /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201112/GNU/
rsync -av /scratch1/BMC/gmtb/Dom.Heinzeller/FV3_RT/TMP_PR274_BASELINE_INTEL/ /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201112/INTEL/

Will let you know when it is done.

climbfuji

I created new baselines on hera.{intel,gnu} and cheyenne.{intel,gnu}, and verified successfully against them. Changes look good to me, but we need to clarify the version number in the next commit at least (see @DusanJovic-NOAA's comments).

climbfuji · 2020-11-17T14:23:35Z

Thanks, Dom. I will copy the baseline to hera. My orion job has not finished yet.
…

No problem. I already started copying the baseline directories on hera, but it's very slow again:
rsync -av /scratch1/BMC/gmtb/Dom.Heinzeller/FV3_RT/TMP_PR274_BASELINE_GNU/ /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201112/GNU/
rsync -av /scratch1/BMC/gmtb/Dom.Heinzeller/FV3_RT/TMP_PR274_BASELINE_INTEL/ /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20201112/INTEL/
Will let you know when it is done.

This is done. I am going to delete the temporary baselines now.

aerorahul

Need tag of hpc-stack and not branches from some individual developers.

aerorahul · 2020-11-17T14:57:44Z

modulefiles/hera.gnu/fv3

@@ -15,13 +15,13 @@ setenv CMAKE_CXX_COMPILER mpicxx
 setenv CMAKE_Fortran_COMPILER mpif90
 setenv CMAKE_Platform hera.gnu

-module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
-
+module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack


Here and elsewhere, please use a tagged version from hpc-stack before merging this in develop

It looks to me that this is an issue with NCEPLIBS group, not the model. If there is an requirement that hpc-stack needs to have version number, then nceplibs group needs to know it and follow that. We need to get the GFS.v16 merged to develop branch so people can start further development. This is the only hpc stack lib provided to us. We can have a meeting with NCEPLIBS group asking them to accept the rule about version requirement, when they come out a new hpc-stack version, we will update it in the model. But so far we have tested this hpc stack library in the model and it is working and we can not afford further delay.

aerorahul · 2020-11-17T14:58:02Z

modulefiles/hera.intel/fv3

@@ -15,13 +15,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc
 setenv CMAKE_Fortran_COMPILER mpiifort
 setenv CMAKE_Platform hera.intel

-module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
+module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack


need a tag.

aerorahul · 2020-11-17T14:58:10Z

modulefiles/hera.intel/fv3_debug

@@ -15,13 +15,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc
 setenv CMAKE_Fortran_COMPILER mpiifort
 setenv CMAKE_Platform hera.intel

-module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
+module use /scratch2/NCEPDEV/nwprod/hpc-stack/test/modulefiles/stack


need a tag.

aerorahul · 2020-11-17T14:58:30Z

modulefiles/orion.intel/fv3

@@ -14,13 +14,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc
 setenv CMAKE_Fortran_COMPILER mpiifort
 setenv CMAKE_Platform orion.intel

-module use /apps/contrib/NCEP/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
+module use /apps/contrib/NCEP/test/hpc-stack-nco/modulefiles/stack


need a tag from develop

aerorahul · 2020-11-17T14:59:52Z

modulefiles/wcoss_dell_p3/fv3

@@ -18,13 +18,13 @@ setenv CMAKE_CXX_COMPILER mpiicpc
 setenv CMAKE_Fortran_COMPILER mpiifort
 setenv CMAKE_Platform wcoss_dell_p3

-module use /usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/v1.0.0-beta1/modulefiles/stack
+module use /usrx/local/nceplibs/dev/hpc-stack/test/hpc-stack/modulefiles/stack


NCEPLIBS team decided on using tags and versions with x.y.z and not using test or branches.

@aerorahul would you please open an issue in NCEPLIBS about this requirement?

DomHeinzeller · 2020-11-17T17:37:50Z

For my planning, can we get started with merging the PRs, or do the modulefile comments (use proper versions instead of test) need to be addressed first?

junwang-noaa · 2020-11-17T18:05:37Z

@aerorahul Please bring the issue with NCEPLIBS team about the requirement. Model only uses the hpc-stack lib provided to us.

aerorahul · 2020-11-17T21:24:52Z

@junwang-noaa I removed my block.

junwang-noaa · 2020-11-17T21:40:10Z

*@aerorahul* this PR still shows " [image: @aerorahul] <https://github.com/aerorahul> aerorahul requested changes ".

…

On Tue, Nov 17, 2020 at 4:25 PM Rahul Mahajan ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> I removed my block. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TM2BNCLLBO5Q2QFQ3TSQLS3HANCNFSM4TTTQ4JA> .

aerorahul · 2020-11-17T21:41:43Z

@aerorahul this PR still shows " [image: @aerorahul] https://github.com/aerorahul aerorahul requested changes ".
…
On Tue, Nov 17, 2020 at 4:25 PM Rahul Mahajan @.***> wrote: @junwang-noaa https://github.com/junwang-noaa I removed my block. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TM2BNCLLBO5Q2QFQ3TSQLS3HANCNFSM4TTTQ4JA .

Not to me.
I dismissed my review.

climbfuji · 2020-11-17T21:48:52Z

@aerorahul this PR still shows " [image: @aerorahul] https://github.com/aerorahul aerorahul requested changes ".
…
On Tue, Nov 17, 2020 at 4:25 PM Rahul Mahajan @.***> wrote: @junwang-noaa https://github.com/junwang-noaa I removed my block. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#274 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI7D6TM2BNCLLBO5Q2QFQ3TSQLS3HANCNFSM4TTTQ4JA .

Not to me.
I dismissed my review.

I think @DusanJovic-NOAA needs to approve (his review still says "requesting changes").

climbfuji · 2020-11-17T21:55:17Z

Hooray!

…es and add support for global/regional grib2 functionality in chgres_cube (#274) * Remove all references to /lfs3 on Jet * Add Ben and Ratko to the CODEOWNERS file * Replace hard-coded make_orog module file with build-level module file in UFS_UTILS * Remove hard-coded make_sfc_climo module file * Add changes for merged chgres_cube code * Add changes for merged chgres_cube code * Minor tweak to FCST_LEN_HRS default in config.community.sh * Changes to make the release version of chgres_cube run in regional_workflow * Changes for regional_grid build on Jet * Changes to regional_grid build for Hera * Change regional_grid makefile for hera * Remove leading zero from FCST_LEN_HRS in config.community.sh * Remove /sorc directory * Remove build module files for codes originally in the regional_workflow repository. Remove run-time make_grid module file for all platforms. Will be sourced from UFS_UTILS from now on. * Update regional grid template for newest code * Copy make_grid module file from UFS_UTILS * Add make_grid.local file for most platforms * Remove alpha and kappa parameters from the regional_grid namelist * Modifications to file type conventions in the chgres_cube namelist for FV3GFS and GSMGFS nemsio files * Set convert_nst=False for global grib2 FV3GFS files when running chgres_cube * Add tracers back into nemsio file processing * Changes to the make_lbcs ex-script (remove all surface-related variables) * Fix for modulefiles * Fixes after merging authoritative repo into fork * Add Thompson climo to chgres_cube namelist for appropriate external model/SDF combinations * Commit new locations for Thompson climo fix file * Change FIXsar to FIXLAM * Change gfs_bndy.nc to gfs.bndy.nc * Move file * Bug fixes to setup.sh and exregional_make_ics.sh * Add support for NAM grib2 files * Path fix * Typo fix * Fix extension on UPP grib2 files * Bug fix for if statement * Add .grib2 extension to soft links * Fix nsoill_out values based on LSM scheme in CCPP suite * Fix grib2 extensions * Add if statement for varmap tables when using Thompson MP and initializing from non-RAP/HRRR data * Final modifications to support NAM grib2 files in regional_workflow * Set climo as default for soil variables when using HRRRX (user will need change this if they know these variables are available for the dates they are running). * Add FV3_CPT_v0 to varmap if statement * Changes to post file names to make model lowercase and move ${fhr} to three values instead of two * Change "rrfs" to "${NET}" instead * Revert "Add FV3_CPT_v0 to varmap if statement" This reverts commit b04ad0b3c8c554f664c6790030a4f33b5a395023. * Add HALO_BLEND back into config_defaults.sh * Set do_deep=false for RRFS_v1beta and other RAP/HRRR suites * Fix if statement for lsoil variable in generate step. * Remove link to fixed file in UFS_UTILS source code directory, now points to FIXgsm * Typo in config_defaults.sh and remove unused SDFs from the make_ics/lbcs scripts, from Gerard's PR * Add FV3_CPT_v0 to list of SDFs that use the GFS varmap table * Remove wgrib2 module load from local make_ics/lbcs files * Use correct system calls for setting up Cheyenne python environment now that these are proper modulefiles Co-authored-by: Michael Kavulich, Jr <kavulich@ucar.edu>

junwang-noaa and others added 11 commits October 15, 2020 15:00

bring GFS.v16 code changes back to develop branch

8b309e2

update upp lib in hera module

36e928c

use test hpc stack lib

5b4a845

update upp lib in fv3 for cmake

796bec2

udpate fv3 to the latest develop branch

32a4352

Merge remote-tracking branch 'upstream/develop' into gfsv16_20201015

ef6b823

update post control files

eb1460b

update upp lib in hpcstack on hera, orion and dell and in nceplibs on…

22b4299

… cray

add change in fv3 dycore ufs-community#35

8a586fc

update FV3 to the develop branch

5113ed8

Merge remote-tracking branch 'upstream/develop' into gfsv16_20201015

09d25b6

This was referenced Nov 12, 2020

Bring GFS.v16 features back to develop branch NOAA-EMC/fv3atm#196

Merged

add GFSv16 dzmin change and changes for Dev/jcsda NOAA-EMC/GFDL_atmos_cubed_sphere#44

Merged

junwang-noaa requested review from DusanJovic-NOAA, climbfuji, WenMeng-NOAA and Hang-Lei-NOAA November 12, 2020 19:34

WenMeng-NOAA approved these changes Nov 12, 2020

View reviewed changes

remove duplicate read_data in fms_mod in external_ic.F90

0c1f2b5

DusanJovic-NOAA reviewed Nov 13, 2020

View reviewed changes

DusanJovic-NOAA requested a review from aerorahul November 13, 2020 14:48

udpate library for hera.gnu

401a357

junwang-noaa mentioned this pull request Nov 13, 2020

use new nceplibs hpc stack libraries in CI test #277

Closed

junwang-noaa and others added 3 commits November 13, 2020 13:51

update FV3 to the top of develop branch; skip-ci

c9e18df

Merge remote-tracking branch 'upstream/develop' into gfsv16_20201015

f8fd3b9

update hera.gnu hpc stack; skip-ci

c2cede8

climbfuji mentioned this pull request Nov 16, 2020

Update module files for Cheyenne to use hpc-stack, final GFS v16 updates in CCPP, revert ccpp_prebuild.py call changee #238

Closed

Merge pull request #5 from climbfuji/gfsv16_20201015

f3544dc

gfsv16_20201015: modulefiles and regression test logs for cheyenne.{intel,gnu} and hera.{intel,gnu}

climbfuji approved these changes Nov 17, 2020

View reviewed changes

update orion log files

8118efe

junwang-noaa added Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. Waiting for Reviews The PR is waiting for reviews from associated component PR's. labels Nov 17, 2020

aerorahul requested changes Nov 17, 2020

View reviewed changes

point fv3 to EMC develop branch

1553c0c

aerorahul self-requested a review November 17, 2020 21:16

aerorahul requested review from aerorahul and removed request for aerorahul November 17, 2020 21:25

aerorahul approved these changes Nov 17, 2020

View reviewed changes

DusanJovic-NOAA approved these changes Nov 17, 2020

View reviewed changes

junwang-noaa merged commit b771e5b into ufs-community:develop Nov 17, 2020

This was referenced Nov 18, 2020

Update ufs-weather-model to the latest hash #284

Closed

need to update checkout.sh to use the latest model hash NOAA-EMC/global-workflow#189

Closed

junwang-noaa deleted the gfsv16_20201015 branch April 20, 2021 18:24

pjpegion pushed a commit to NOAA-PSL/ufs-weather-model.p7b that referenced this pull request Jul 20, 2021

Fix submodule pointer for ccpp-physics (ufs-community#274)

96ae478

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bring GFS.v16 features back to develop branch #274

Bring GFS.v16 features back to develop branch #274

junwang-noaa commented Nov 12, 2020 •

edited

Loading

WenMeng-NOAA left a comment

WenMeng-NOAA commented Nov 12, 2020

junwang-noaa commented Nov 12, 2020 via email

WenMeng-NOAA commented Nov 12, 2020 via email

DusanJovic-NOAA Nov 13, 2020

junwang-noaa commented Nov 13, 2020 via email

junwang-noaa commented Nov 17, 2020 via email

climbfuji commented Nov 17, 2020

climbfuji left a comment

climbfuji commented Nov 17, 2020

aerorahul left a comment

aerorahul Nov 17, 2020

junwang-noaa Nov 17, 2020

aerorahul Nov 17, 2020

aerorahul Nov 17, 2020

aerorahul Nov 17, 2020

aerorahul Nov 17, 2020

junwang-noaa Nov 17, 2020

DomHeinzeller commented Nov 17, 2020

junwang-noaa commented Nov 17, 2020

aerorahul commented Nov 17, 2020

junwang-noaa commented Nov 17, 2020 via email

aerorahul commented Nov 17, 2020

climbfuji commented Nov 17, 2020 •

edited

Loading

climbfuji commented Nov 17, 2020

Bring GFS.v16 features back to develop branch #274

Bring GFS.v16 features back to develop branch #274

Conversation

junwang-noaa commented Nov 12, 2020 • edited Loading

Description

Issue(s) addressed

Testing

Dependencies

WenMeng-NOAA left a comment

Choose a reason for hiding this comment

WenMeng-NOAA commented Nov 12, 2020

junwang-noaa commented Nov 12, 2020 via email

WenMeng-NOAA commented Nov 12, 2020 via email

Choose a reason for hiding this comment

junwang-noaa commented Nov 13, 2020 via email

junwang-noaa commented Nov 17, 2020 via email

climbfuji commented Nov 17, 2020

climbfuji left a comment

Choose a reason for hiding this comment

climbfuji commented Nov 17, 2020

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DomHeinzeller commented Nov 17, 2020

junwang-noaa commented Nov 17, 2020

aerorahul commented Nov 17, 2020

junwang-noaa commented Nov 17, 2020 via email

aerorahul commented Nov 17, 2020

climbfuji commented Nov 17, 2020 • edited Loading

climbfuji commented Nov 17, 2020

junwang-noaa commented Nov 12, 2020 •

edited

Loading

climbfuji commented Nov 17, 2020 •

edited

Loading