Merge pull request #123 from grantfirl/ufs-dev-PR139

UFS-dev PR#139

.github/pull_request_template.md

@@ -1,32 +1,56 @@
-<!-- THE FOLLOWING IS FOR THE PR AUTHOR TO FILL OUT
-PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
-## PR Author Checklist:
-<!--  Please complete all items in list. -->
-- [ ] I have linked PR's from all sub-components involved in section below. <!-- PLEASE DO NOT LINK SUBCOMPONENT ISSUES -->
-- [ ] I am confirming reviews are completed in ALL sub-component PR's.
-- [ ] I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
-  - LOG: 
-- [ ] I have added the list of all failed regression tests to "Anticipated changes" section.
-- [ ] I have filled out all sections of the template.
-
-## Description
+<!-- INSTRUCTIONS: 
+- Please fill out all sections of this PR and complete the checklist below
+- Please be as descriptive as possible, this is really important.
+- Please "fill in" checkboxes. Use [X] for a filled in checkbox or leave it [ ] for an empty checkbox
+- Please use github markup as much as possible in linking
+i.e.:
+* Linking to UFSWM PR's and issues add - #<pr/issue number>
+* Linking to a subcomponent PR and issues add - <Group>/<Fork>/pull/<number> or - <Group>/<Fork>/issues/<number>
+-->
+## Commit Queue Requirements:
+<!-- Please note: PRs will be scheduled in the Commit Queue in the order received and only after all
+pre-requisite testing is complete and all PR requirements (e.g. Issues created and noted, Subcomponent
+PRs reviewed and accepted) are met. -->
+- [ ] Fill out all sections of this template.
+- [ ] All sub component pull requests have been reviewed by their code managers.
+- [ ] Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
+- [ ] Add list of all failed regression tests in "Regression Tests" section.
+
+## PR Information
+
+### Description
 <!-- Provide a detailed description of what this PR does in the space provided below-->
 
+### Commit Message
+<!--
+Please provide the following concise information:
+Description of all changes - 1 line
+Please list all individual issue titles addressed with github links at the end in parenthesis (using #<number> or <group>/<fork>/issues/<number>).
+-->
 
-## Linked Issues and Pull Requests
-### Associated UFSWM Issue to close
-<!-- Example: "- Closes #1698" -->
 
+### Priority
+- [ ] Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
+- [ ] High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
+- [ ] Normal
 
-### Subcomponent Pull Requests
-<!-- format: - <community>/<repo>/pull/<PR number> i.e.: - NOAA-EMC/fv3atm/pull/33 or "None" -->
+### Blocking Dependencies
+<!-- If there are any PR's that are needed to be completed before this one, please add links
+     to them here -->
 
+### Git Issues Fixed By This PR
+<!-- Example: - Closes #1698  or - Closes NOAA-EMC/fv3atm/issues/729 -->
 
-### Blocking Dependencies
-<!-- Example: "- Depends on #1733" or "None" -->
 
+## Changes
 
-### Subcomponents involved:
+### Subcomponent (with links)
+<!-- (add links to subcomponent PR's here) -->
+<!-- Example:
+[X] FV3
+- NOAA-EMC/fv3atm/pull/734
+- NOAA-EMC/fv3atm/pull/735
+-->
 - [ ] AQM
 - [ ] CDEPS
 - [ ] CICE
@@ -41,7 +65,6 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
 - [ ] stochastic_physics
 - [ ] none
 
-## Anticipated Changes
 ### Input data
 - [ ] No changes are expected to input data.
 - [ ] Changes are expected to input data:
@@ -51,9 +74,8 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
 ### Regression Tests:
 - [ ] No changes are expected to any regression test.
 - [ ] Changes are expected to the following tests:
-<!-- Please insert what RT's change and why you expect them to change in the space provided below -->
-<details><summary>Tests effected by changes in this PR:</summary>
-<!-- ADD ITEMS HERE or add "None" -->
+<details><summary>FAILED REGRESSION TESTS</summary>
+<!-- List failed regression tests here or add "None" -->
 
 </details>
 
@@ -64,28 +86,20 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
   - [ ] Create separate issue in [JCSDA/spack-stack](https://github.com/JCSDA/spack-stack) asking for update to library. Include library name, library version.
   - [ ] Add issue link from JCSDA/spack-stack following this item <!-- for example: "- JCSDA/spack-stack/issue/1757" -->
 
-
-<!-- THE FOLLOWING IS FOR CODE MANAGERS ONLY DO NOT FILL OUT -->
-<details><summary>Code Managers Log</summary>
-
-- [ ] This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
-- [ ] Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
-  - [ ] N/A
-
+<!-- STOP!!! THE FOLLOWING IS FOR CODE MANAGERS ONLY. PLEASE DO NOT FILL OUT -->
 ### Testing Log:
 - RDHPCS
   - [ ] Hera
   - [ ] Orion
   - [ ] Hercules
   - [ ] Jet
   - [ ] Gaea
-  - [ ] Cheyenne
+  - [ ] Derecho
 - WCOSS2
   - [ ] Dogwood/Cactus
   - [ ] Acorn
 - CI
   - [ ] Completed
 - opnReqTest
   - [ ] N/A
-  - [ ] Log attached to comment
-</details>
+  - [ ] Log attached to comment

cmake/configure_gaea-c5.intel.cmake

@@ -0,0 +1,4 @@
+set(PARALLEL_NETCDF ON  CACHE BOOL "Enable parallel NetCDF" FORCE)
+set(MOM6_Extra_FORTRAN_FLAGS "-xsse2")
+set(HYCOM_Extra_FORTRAN_FLAGS "-xSSE4.2")
+set(HYCOM_Extra_C_FLAGS "-xSSE4.2")

doc/UsersGuide/source/AutomatedTesting.rst

@@ -91,7 +91,7 @@ On supported HPC systems, a :term:`cron job` runs the ``start_rt_auto.sh`` bash
 This script checks the HPC name and sets certain python paths. Then, it runs ``rt_auto.py``, 
 which uses the Github API (through pyGitHub) to check the labels on pull requests to 
 ``ufs-weather-model``. If a PR label matches the HPC name 
-(e.g., hera-intel-RT or cheyenne-gnu-BL), the label provides the HPC  
+(e.g., hera-intel-RT or derecho-gnu-BL), the label provides the HPC  
 with the compiler and job information to run a test or task on the machine. 
 If no PR label matches HPC name, the script exits.
 

doc/UsersGuide/source/BuildingAndRunning.rst

@@ -10,7 +10,7 @@ Supported Platforms & Compilers
 Before running the Weather Model (:term:`WM`), users should determine which of the 
 :ref:`levels of support <SupportedPlatforms>` 
 is applicable to their system. Generally, Level 1 & 2 systems are restricted to those with access 
-through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Cheyenne). 
+through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Derecho). 
 Level 3 & 4 systems include certain personal computers or non-NOAA-affiliated HPC systems. 
 The prerequisite software libraries for building the WM already exist in a centralized location on Level 1/preconfigured 
 systems, so users may skip directly to :ref:`getting the data <GetData>` and downloading the code. 
@@ -89,23 +89,23 @@ the data required to run the WM RTs are already available in the following locat
 .. _DataLocations:
 .. table:: Data Locations for Level 1 & 2 Systems
 
-   +--------------+-----------------------------------------------------+
-   | Machine      | File location                                       |
-   +==============+=====================================================+
-   | Cheyenne     | /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT  |
-   +--------------+-----------------------------------------------------+
-   | Gaea         | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT        |
-   +--------------+-----------------------------------------------------+
-   | Hera         | /scratch1/NCEPDEV/nems/emc.nemspara/RT              |
-   +--------------+-----------------------------------------------------+
-   | Jet          | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT                 |
-   +--------------+-----------------------------------------------------+
-   | Orion        | /work/noaa/nems/emc.nemspara/RT                     |
-   +--------------+-----------------------------------------------------+
-   | S4           | /data/prod/emc.nemspara/RT                          |
-   +--------------+-----------------------------------------------------+ 
-   | WCOSS2       | /lfs/h2/emc/nems/noscrub/emc.nems/RT                |
-   +--------------+-----------------------------------------------------+ 
+   +--------------+--------------------------------------------------------+
+   | Machine      | File location                                          |
+   +==============+========================================================+
+   | Derecho      | /glade/derecho/scratch/epicufsrt/ufs-weather-model/RT  |
+   +--------------+--------------------------------------------------------+
+   | Gaea         | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT           |
+   +--------------+--------------------------------------------------------+
+   | Hera         | /scratch1/NCEPDEV/nems/emc.nemspara/RT                 |
+   +--------------+--------------------------------------------------------+
+   | Jet          | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT                    |
+   +--------------+--------------------------------------------------------+
+   | Orion        | /work/noaa/nems/emc.nemspara/RT                        |
+   +--------------+--------------------------------------------------------+
+   | S4           | /data/prod/emc.nemspara/RT                             |
+   +--------------+--------------------------------------------------------+ 
+   | WCOSS2       | /lfs/h2/emc/nems/noscrub/emc.nems/RT                   |
+   +--------------+--------------------------------------------------------+ 
 
 For Level 3-4 systems, the data must be added to the user's system. 
 Publicly available RT data is available in the `UFS WM Data Bucket <https://registry.opendata.aws/noaa-ufs-regtests/>`__. 

doc/UsersGuide/source/FAQ.rst

@@ -29,14 +29,14 @@ The steps are:
          cd tests
 
    #. Find a configure (``*.conf``) file that contains the machine and compiler you are using. For this
-      example, the Intel compiler on Cheyenne is used.  To create a custom configure file, two lines are
+      example, the Intel compiler on Derecho is used.  To create a custom configure file, two lines are
       needed:  a ``COMPILE`` line and a ``RUN`` line.   The ``COMPILE`` line should contain the name
-      of the machine and compiler ``cheyenne.intel`` and the desired ``SUITES`` for the build.  Choose a
+      of the machine and compiler ``derecho.intel`` and the desired ``SUITES`` for the build.  Choose a
       ``RUN`` line under this ``COMPILE`` command that uses the desired ``SUITE``.  For example:
 
       .. code-block:: console
 
-         COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst                     | standard    | cheyenne.intel | fv3
+         COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst                     | standard    | derecho.intel | fv3
          RUN     | fv3_ccpp_gfs_v16beta                                                                                                           | standard    |                | fv3         |
 
       Put these two lines into a file called ``my_test.conf``.  The parameters used in this run can be
@@ -49,16 +49,16 @@ The steps are:
 
       .. code-block:: console
 
-         if [[ $MACHINE_ID = cheyenne.* ]]; then stanza:
+         if [[ $MACHINE_ID = derecho.* ]]; then stanza:
          ...
          dprefix=/glade/scratch
 
-      This works for Cheyenne, since ``$USER/FV3_RT`` will be appended.  Also check that ``RTPWD``
+      This works for Derecho, since ``$USER/FV3_RT`` will be appended.  Also check that ``RTPWD``
       points to a diretory that exists:
 
       .. code-block:: console
 
-         if [[ $MACHINE_ID = cheyenne.* ]]; then
+         if [[ $MACHINE_ID = derecho.* ]]; then
             RTPWD=${RTPWD:-$DISKNM/ufs-public-release-20200224/${COMPILER^^}}
 
    #. Run the ``rt.sh`` script from the ``tests`` directory:

doc/UsersGuide/source/InputsOutputs.rst

@@ -1404,10 +1404,6 @@ shows the following parameters that can be set in ``model_configure`` at run-tim
      - atmosphere time step in second
      - integer
      - 1800 (for C96)
-   * - output_1st_tstep_rst
-     - output first time step history file after restart
-     - logical
-     - .false.
    * - restart_interval
      - frequency to output restart file or forecast hours to write out restart file
      - integer
@@ -1452,22 +1448,6 @@ shows the following parameters that can be set in ``model_configure`` at run-tim
      - j-dimension for output grid
      - integer
      - 190
-   * - nfhout
-     - history file output frequency
-     - integer
-     - 3
-   * - nfhmax_hf
-     - forecast length of high history file
-     - integer
-     - 0 (0:no high frequency output)
-   * - nfhout_hf
-     - high history file output frequency
-     - integer
-     - 1
-   * - nsout
-     - output frequency of number of time step
-     - integer
-     - -1 (negative: turn off the option, 1: output history file at every time step)
    * - output_fh
      - history file output forecast hours or history file output frequency if the second elelment is -1
      - real

doc/UsersGuide/source/Introduction.rst

@@ -36,7 +36,7 @@ for those elements, are listed below:
 
 The UFS Weather Model is currently included in two UFS Application releases: The UFS Short-Range Weather (:term:`SRW`) Application v2.0.0 release (June 2022) and the UFS Medium Range Weather Application (:term:`MRW`) v1.1.0 release (October 2020). These UFS Apps also contain pre- and post-processing components, a comprehensive build system, and workflows for configuration and execution of the application. The SRW App v2.0.0 documentation and details can be found `here <https://ufs-srweather-app.readthedocs.io/en/release-public-v2/>`__. The MRW App v1.1.0 documentation and details can be found `here <https://ufs-mrweather-app.readthedocs.io/en/ufs-v1.1.0>`__.
 
-The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system, the National Science Foundation Stampede system, and Mac laptops.
+The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Derecho system, the National Science Foundation Stampede system, and Mac laptops.
 
 .. note::
 
@@ -66,4 +66,4 @@ This WM User's Guide is organized as follows:
 
 Finally, :numref:`Chapters %s <Acronyms>` and :numref:`%s <Glossary>` contain a list of acronyms and a glossary, respectively.
 
-.. bibliography:: references.bib
+.. bibliography:: references.bib