Merge pull request #224 from climbfuji/myj_with_dom_mods

gmtb/develop: update from master, address issues from previous commit

doc/CCPPtechnical/source/BuildingRunningHostModels.rst

@@ -342,7 +342,7 @@ Regression testing is the process of testing changes to the programs to make sur
 Overview of the RTs
 ^^^^^^^^^^^^^^^^^^^
 
-The RT configuration files are located in ``./tests`` relative to the top-level directory of NEMSfv3gfs and have names ``rt*.conf``. The default RT configuration file, supplied with the NEMSfv3gfs master, compares the results from the non-CCPP code to the *official baseline* and is called ``rt.conf``. Before running the RT script ``rt.sh`` in the same directory, the user has to set one or more environment variables and potentially modify the script to change the location of the automatically created run directories. The environment variables are ``ACCNR`` (mandatory unless the user is a member of the default project *nems*; sets the account to be charged for running the RTs), ``NEMS_COMPILER`` (optional for the ``intel`` compiler option, set to ``gnu`` to switch), and potentially ``RUNDIR_ROOT``. ``RUNDIR_ROOT`` allows the user to specify an alternative location for the RT run directories underneath which directories called ``rt_$PID`` are created (``$PID`` is the process identifier of the ``rt.sh`` invocation). This may be required on systems where the user does not have write permissions in the default run directory tree.
+The RT configuration files are located in ``./tests`` relative to the top-level directory of NEMSfv3gfs and have names ``rt*.conf``. The default RT configuration file, supplied with the NEMSfv3gfs master is called ``rt.conf`` and runs four types of configurations: IPD PROD, IPD REPRO, CCPP PROD, and CCPP REPRO. For the IPD configurations, CCPP is not used, that is, the code is compiled with ``CCPP=N``. The PROD configurations use the compiler flags used in NCEP operations for superior performance, while the REPRO configurations remove certain compiler flags to create b4b identical results between CCPP and IPD configurations. Before running the RT script ``rt.sh`` in directory ``./tests``, the user has to set some environment variables on the working shell: ``ACCNR`` (account to be charged for running the RTs), ``NEMS_COMPILER`` (optional for the ``intel`` compiler option, set to ``gnu`` to switch), and potentially ``RUNDIR_ROOT`` (location for the RT run directories), underneath which directories called ``rt_$PID`` are created (``$PID`` is the process identifier of the ``rt.sh`` invocation). This may be required on systems where the user does not have write permissions in the default run directory tree.
 
 .. code-block:: console
 
@@ -356,15 +356,15 @@ Running the full default RT suite defined in ``rt.conf`` using the script ``rt.s
 
     ./rt.sh -f
 
-This command can only be used on a NOAA machine using the Intel compiler, where the output of a non-CCPP build using the default Intel version is compared against the *official baseline*. For information on testing the CCPP code, or using alternate computational platforms, see the following sections.
+This command can only be used on a NOAA machine using the Intel compiler, where an *official baseline* is available. For information on testing the CCPP code, or using alternate computational platforms, see the following sections.
 
 This command and all others below produce log output in ``./tests/log_machine.compiler``. These log files contain information on the location of the run directories that can be used as templates for the user. Each ``rt*.conf`` contains one or more compile commands preceding a number of tests.
 
 
 Baselines
 ^^^^^^^^^^^^^^^^^^^
 
-Regression testing is only possible on machines for which baselines exist. EMC maintains *official baselines* of non-CCPP runs on *Jet* and *Wcoss* created with the Intel compiler. GMTB maintains additional baselines on *Theia, Jet, Cheyenne*, and *Gaea*. While GMTB is trying to keep up with changes to the official repositories, baselines maintained by GMTB are not guaranteed to be up-to-date.
+Regression testing is only possible on machines for which baselines exist. EMC maintains *official baselines* on *Theia* and *Wcoss* created with the Intel compiler. GMTB maintains additional baselines on *Jet*, *Cheyenne*, and *Gaea*. While GMTB is trying to keep up with changes to the official repositories, baselines maintained by GMTB are not guaranteed to be up-to-date.
 
 When porting the code to a new machine, it is useful to start by establishing a *personal baseline*. Future runs of the RT can then be compared against the *personal baseline* to ascertain that the results have not been inadvertently affected by code developments. The ``rt.sh -c`` option is used to create a *personal baseline*.
 
@@ -404,7 +404,7 @@ The *official baseline* directory is defined as:
     RTPWD=$DISKNM/trunk-yyyymmdd/${COMPILER} # on Cheyenne
     RTPWD=$DISKNM/trunk-yyyymmdd             # elsewhere
 
-Note that ``yyyymmdd`` is the year, month and day the RT was created.
+Note that ``yyyymmdd`` is the year, month and day the baseline was created using top of master code.
 
 .. warning::  Modifying ``$DISKNM`` will break the RTs!
 
@@ -435,25 +435,6 @@ In case a user does not have write permissions to ``$STMP (/scratch4/NCEPDEV/stm
     RUNDIR_ROOT=${RUNDIR_ROOT:-${PTMP}/${USER}/FV3_RT}/rt_$$
 
 
-Non-CCPP vs CCPP Tests
-^^^^^^^^^^^^^^^^^^^^^^
-
-While the official EMC RTs do not execute the CCPP code, GMTB provides RTs to exercise the CCPP in its various modes: ``rt_ccpp_standalone.conf`` tests the CCPP with dynamic build and ``rt_ccpp_static.conf`` tests the CCPP with static build. These tests compare the results of runs done using the CCPP against a previously generated *personal baseline* created without the CCPP by running ``rt_ccpp_ref.conf``.  For this comparison, both the non-CCPP *personal baseline* and the tests using the CCPP are performed with code built with the :term:`REPRO` compiler options.
-
-The command below should be used to create a *personal baseline* using non-CCPP code compiled in :term:`REPRO` mode.
-
-.. code-block:: console
-
-    ./rt.sh -l rt_ccpp_ref.conf -c fv3 # create own reg. test baseline
-
-Once the *personal baseline* in REPRO mode has been created, the CCPP tests can be run to compare against it. Use the ``-l`` option to select the test suite and the ``-m`` option to compare against the *personal baseline*.
-
-.. code-block:: console
-
-    ./rt.sh -l rt_ccpp_standalone.conf -m # dynamic build
-    ./rt.sh -l rt_ccpp_static.conf -m     # static build
-
-
 Compatibility between the Code Base, the SDF, and the Namelist in the UFS Atmosphere
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

doc/CCPPtechnical/source/CodeManagement.rst

@@ -14,7 +14,7 @@ This chapter describes the organization of the code, provides instruction on the
 UFS Atmosphere
 -----------------------
 
-The :term:`UFS` Atmosphere source code is contained in the NEMSfv3gfs code repository. CCPP users and developers should use the NEMSfv3gfs code and submodules maintained by GMTB in GitHub. For codes whose authoritative repository is in VLab, GMTB synchronizes these with VLab at periodic intervals. 
+The :term:`UFS` Atmosphere source code is contained in the NEMSfv3gfs code repository. CCPP users and developers should use the NEMSfv3gfs code and submodules maintained by GMTB in GitHub. For codes whose authoritative repository is managed by EMC, GMTB synchronizes these with EMC's at periodic intervals. Note that EMC is incrementally transitioning its repositories from VLab to GitHub, and therefore this code management is expected to change in the future. 
 
 https://github.com/NCAR/NEMSfv3gfs
 
@@ -32,23 +32,22 @@ https://github.com/NCAR/FMS
 
 Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories.
 
-Some of these repositories are public (no GitHub account required) and some are private. The public repositories (ccpp-framework, ccpp-physics, and Flexible Modeling System - FMS) may be used directly to read or create forks. Write permission is generally restricted, however. The private repositories require access - please send a request and your GitHub username to gmtb-help@ucar.edu.
-
-The primary development by GMTB, including the latest CCPP developments, are maintained in the following branches:
+Some of these repositories are public (no GitHub account required) and some are private. The public repositories (ccpp-framework, ccpp-physics, NEMS, and Flexible Modeling System - FMS) may be used directly to read or create forks. Write permission is generally restricted, however. The private repositories require access - please send a request and your GitHub username to gmtb-help@ucar.edu.
 
+The following branches are recommended for CCPP users and developers:
 
 +---------------------------------------------+----------------------+
 | Repository (GMTB development version)       |    Branch name       |
 +=============================================+======================+
-| https://github.com/NCAR/NEMSfv3gfs          |    gmtb/ccpp         |
+| https://github.com/NCAR/NEMSfv3gfs          |   master             |
 +---------------------------------------------+----------------------+
-| https://github.com/NCAR/FV3                 |    gmtb/ccpp         |
+| https://github.com/NCAR/FV3                 |   master             |
 +---------------------------------------------+----------------------+
 | https://github.com/NCAR/ccpp-physics        |   master             |
 +---------------------------------------------+----------------------+
 | https://github.com/NCAR/ccpp-framework      |   master             |
 +---------------------------------------------+----------------------+
-| https://github.com/NCAR/NEMS                |   gmtb/ccpp          |
+| https://github.com/NCAR/NEMS                |   develop            |
 +---------------------------------------------+----------------------+
 | https://github.com/NCAR/FMS                 |    GFS-FMS           |
 +---------------------------------------------+----------------------+
@@ -61,15 +60,15 @@ CCPP developers should use the SCM code and submodules maintained by GMTB in Git
 
 https://github.com/NCAR/gmtb-scm
 
-As for NEMSfv3gfs, there are two submodules referenced in the gmtb-scm repository:
+As with NEMSfv3gfs, there are two submodules referenced in the gmtb-scm repository:
 
 https://github.com/NCAR/ccpp-framework
 
 https://github.com/NCAR/ccpp-physics
 
-Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories. Both CCPP repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however. The SCM repository is private, to request access please send a message and your GitHub username to gmtb-help@ucar.edu.
+Users have read-only access to these repositories and as such cannot accidentally destroy any important (shared) branches of these authoritative repositories. Both CCPP repositories and the SCM repositories are public (no GitHub account required) and may be used directly to read or create forks. Write permission is generally restricted, however. 
 
-The primary development by GMTB, including the latest CCPP developments, are maintained in the following branches:
+The following branches are recommended for CCPP users and developers:
 
 +----------------------------------------+-------------------+
 | Repository (GMTB development version)  | Branch name       |
@@ -162,7 +161,7 @@ Start with checking out the main repository from the NCAR GitHub
 
 .. code-block:: console
 
-   git clone -b gmtb/ccpp https://github.com/NCAR/NEMSfv3gfs
+   git clone https://github.com/NCAR/NEMSfv3gfs
    cd NEMSfv3gfs
    git submodule init
    git submodule update
@@ -181,7 +180,7 @@ Checking out remote branches as submodules means that your local branches are in
 
    cd NEMS
    git remote update
-   git checkout upstream/gmtb/ccpp
+   git checkout upstream/develop
    cd ..
 
 However, if you are making changes in a repository (submodule or main repository), you must create a local branch, for example in NEMSfv3gfs:
@@ -217,7 +216,7 @@ As opposed to branches without modifications described in step 3, changes to the
 
    cd FV3
    git remote update
-   git pull upstream gmtb/ccpp
+   git pull upstream master
 
 
 -----------------------------------
@@ -371,6 +370,8 @@ Go to the github.com web interface, and navigate to your repository fork and bra
  | Fill in a detailed description, including reporting on any testing you did
  | Click on “Create pull request”
 
+If your development also requires changes in other repositories, you must open PRs in those repositories as well. In the PR message for each repository, please note the associate PRs submitted to other repositories.
+
 Several people (aka CODEOWNERS) are automatically added to the list of reviewers on the right hand side. If others should be reviewing the code, click on the “reviewers” item on the right hand side and enter their GitHub usernames
 
 Once the PR has been approved, the change is merged to master by one of the code owners. If there are pending conflicts, this means that the code is not up to date with the trunk. To resolve those, pull the target branch from upstream as described above, solve the conflicts and push the changes to the branch on your fork (this also updates the PR).

doc/CCPPtechnical/source/CompliantPhysicsParams.rst

@@ -126,7 +126,7 @@ Metadata Rules
 
 * ``<type>`` can be ``scheme``, ``module``, ``DDT``, or ``host``.
 
-* For empty schemes, the three lines above are sufficient. For non-empty schemes, the metadata should
+* For empty schemes, the three lines above are sufficient. For non-empty schemes, the metadata must
   describe all input and output arguments to the scheme using the following format:
 
 .. code-block:: fortran

doc/CCPPtechnical/source/ConstructingSuite.rst

@@ -8,7 +8,7 @@ Constructing Suites
 Suite Definition File
 ==============================
 
-The :term:`SDF` is a file in XML format used to specify the name of the suite, the physics schemes to run, groups of physics that run together, the order in which to run the physics, and whether subcycling will be used to run any of the parameterizations with shorter timesteps. The :term:`SDF` files are located in ``ccpp/suites/suite_*.xml``.
+The :term:`SDF` is a file in XML format used to specify the name of the suite, the physics schemes to run, groups of physics that run together, the order in which to run the physics, and whether subcycling will be used to run any of the parameterizations with shorter timesteps. The :term:`SDF` files are part of the host model code.
 
 In addition to the primary parameterization categories (such as radiation, boundary layer, deep convection, resolved moist physics, etc.), the :term:`SDF` can have an arbitrary number of interstitial schemes in between the parameterizations to preprocess or postprocess data. In many models, this interstitial code is not obvious to the model user but, with the :term:`SDF`, both the primary parameterizations and the interstitial schemes are listed explicitly.
 

scripts/metadata_table.py

@@ -97,7 +97,7 @@
 from metavar     import Var, VarDictionary
 from parse_tools import ParseObject, ParseSource, register_fortran_ddt_name
 from parse_tools import ParseInternalError, ParseSyntaxError, CCPPError
-from parse_tools import LITERAL, FORTRAN_ID, FORTRAN_SCALAR_REF
+from parse_tools import LITERAL_INT, FORTRAN_ID, FORTRAN_SCALAR_REF
 from parse_tools import check_fortran_ref
 
 ########################################################################
@@ -172,7 +172,7 @@ class MetadataHeader(ParseSource):
 
     __header_start__ = re.compile(r"(?i)\s*\[\s*ccpp-arg-table\s*\]")
 
-    __var_start__ = re.compile(r"^\[\s*("+FORTRAN_ID+r"|"+LITERAL+r"|"+FORTRAN_SCALAR_REF+r")\s*\]$")
+    __var_start__ = re.compile(r"^\[\s*("+FORTRAN_ID+r"|"+LITERAL_INT+r"|"+FORTRAN_SCALAR_REF+r")\s*\]$")
 
     __blank_line__ = re.compile(r"\s*[#;]")
 

scripts/metavar.py

@@ -293,8 +293,7 @@ def valid_value(self, test_value, error=False):
         elif self.type is bool:
             if isinstance(test_value, str):
                 if test_value.lower() in VariableProperty.__true_vals + VariableProperty.__false_vals:
-                    valid_val = (test_value.lower() in VariableProperty.__true_vals) or \
-                                not (test_value.lower() in VariableProperty.__false_vals)
+                    valid_val = test_value.lower() in VariableProperty.__true_vals
                 else:
                     valid_val = None # i.e., pass
             else:
@@ -370,12 +369,7 @@ class Var(object):
                     VariableProperty('dimensions', list,
                                      check_fn_in=check_dimensions),
                     VariableProperty('type', str,
-                                     # DH* Avoid checking of DDTs being registered or not,
-                                     # this only works if all DDTs required in one metadata
-                                     # are defined in that file and/or parsed beforehand.
-                                     #check_fn_in=check_fortran_type),
-                                     # *DH
-                                     ),
+                                     check_fn_in=check_fortran_type),
                     VariableProperty('kind', str,
                                      optional_in=True,
                                      default_fn_in=default_kind_val),

scripts/parse_tools/__init__.py

@@ -32,7 +32,7 @@
 from parse_source   import ParseSyntaxError, ParseInternalError
 from parse_source   import CCPPError, context_string
 from parse_object   import ParseObject
-from parse_checkers import check_fortran_id, LITERAL, FORTRAN_ID
+from parse_checkers import check_fortran_id, LITERAL_INT, FORTRAN_ID
 from parse_checkers import FORTRAN_DP_RE
 from parse_checkers import check_fortran_ref, FORTRAN_SCALAR_REF
 from parse_checkers import check_fortran_intrinsic

scripts/parse_tools/parse_checkers.py

@@ -127,15 +127,15 @@ def check_cf_standard_name(test_val, error=False):
 
 ########################################################################
 
-# LITERAL is a strin representing a literal value
-LITERAL = r"([0-9]*)"
+# LITERAL_INT is a string representing an integer value
+LITERAL_INT = r"([0-9]*)"
 # FORTRAN_ID is a string representing the regular expression for Fortran names
 FORTRAN_ID = r"([A-Za-z][A-Za-z0-9_]*)"
 __FID_RE = re.compile(FORTRAN_ID+r"$")
 # Note that the scalar array reference expressions below are not really for
 # scalar references because a colon can be a placeholder, unlike in Fortran code
-FORTRAN_SCALAR_ARREF = r"\(\s*(?:"+FORTRAN_ID+r"|"+LITERAL+r"|[:])\s*(?:,\s*(?:"+FORTRAN_ID+r"|"+LITERAL+r"|[:])\s*){0,6}\)"
-FORTRAN_SCALAR_REF = r"(?:"+"(?:"+LITERAL+r"|"+FORTRAN_ID+")"+r"\s*"+"(?:"+LITERAL+r"|"+FORTRAN_SCALAR_ARREF+r")"+r")"
+FORTRAN_SCALAR_ARREF = r"\(\s*(?:"+FORTRAN_ID+r"|"+LITERAL_INT+r"|[:])\s*(?:,\s*(?:"+FORTRAN_ID+r"|"+LITERAL_INT+r"|[:])\s*){0,6}\)"
+FORTRAN_SCALAR_REF = r"(?:"+"(?:"+LITERAL_INT+r"|"+FORTRAN_ID+")"+r"\s*"+"(?:"+LITERAL_INT+r"|"+FORTRAN_SCALAR_ARREF+r")"+r")"
 _FORTRAN_SCALAR_REF_RE = re.compile(FORTRAN_SCALAR_REF+r"$")
 FORTRAN_INTRINSIC_TYPES = [ "integer", "real", "logical", "complex",
                             "double precision", "character" ]
@@ -354,13 +354,15 @@ def check_fortran_type(typestr, error=False):
         match = registered_fortran_ddt_name(typestr)
         dt = " derived"
     # End if
-    if match is None:
-        if error:
-            raise CCPPError("'{}' is not a valid{} Fortran type".format(typestr, dt))
-        else:
-            typestr = None
-        # End if
-    # End if
+    # DH* 20190913 - skip checking if a DDT is registered at this time
+    #if match is None:
+    #    if error:
+    #        raise CCPPError("'{}' is not a valid{} Fortran type".format(typestr, dt))
+    #    else:
+    #        typestr = None
+    #    # End if
+    ## End if
+    # *DH 20190913
     return typestr
 
 ########################################################################