Feature 1764 point stat

met/data/config/PointStatConfig_default

@@ -280,6 +280,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE; // Only for HiRA.
+   orank  = NONE; // Only for HiRA.
    rps    = NONE; // Only for HiRA.
    eclv   = NONE;
    mpr    = NONE;

met/docs/Users_Guide/grid-stat.rst

@@ -388,7 +388,7 @@ The **output_flag** array controls the type of output that the Grid-Stat tool ge
 
 10. **VAL1L2** for Vector Anomaly L1L2 Partial Sums when climatological data is supplied
 
-11. **VCNT** for Vector Contingency Table Statistics
+11. **VCNT** for Vector Continuous Statistics
 
 12. **PCT** for Contingency Table Counts for Probabilistic forecasts
 

met/docs/Users_Guide/point-stat.rst

@@ -121,7 +121,7 @@ The forecast value at P is chosen as the grid point inside the interpolation are
 HiRA framework
 ~~~~~~~~~~~~~~
 
-The Point-Stat tool has been enhanced to include the High Resolution Assessment (HiRA) verification logic (:ref:`Mittermaier, 2014 <Mittermaier-2014>`). HiRA is analogous to neighborhood verification but for point observations. The HiRA logic interprets the forecast values surrounding each point observation as an ensemble forecast. These ensemble values are processed in two ways. First, the ensemble continuous statistics (ECNT) and the ranked probability score (RPS) line types are computed directly from the ensemble values. Second, for each categorical threshold specified, a fractional coverage value is computed as the ratio of the nearby forecast values that meet the threshold criteria. Point-Stat evaluates those fractional coverage values as if they were a probability forecast. When applying HiRA, users should enable the matched pair (MPR), probabilistic (PCT, PSTD, PJC, or PRC), continuous ensemble statistics (ECNT), or ranked probability score (RPS) line types in the **output_flag** dictionary. The number of probabilistic HiRA output lines is determined by the number of categorical forecast thresholds and HiRA neighborhood widths chosen.
+The Point-Stat tool has been enhanced to include the High Resolution Assessment (HiRA) verification logic (:ref:`Mittermaier, 2014 <Mittermaier-2014>`). HiRA is analogous to neighborhood verification but for point observations. The HiRA logic interprets the forecast values surrounding each point observation as an ensemble forecast. These ensemble values are processed in three ways. First, the ensemble continuous statistics (ECNT), the observation rank statistics (ORANK) and the ranked probability score (RPS) line types are computed directly from the ensemble values. Second, for each categorical threshold specified, a fractional coverage value is computed as the ratio of the nearby forecast values that meet the threshold criteria. Point-Stat evaluates those fractional coverage values as if they were a probability forecast. When applying HiRA, users should enable the matched pair (MPR), probabilistic (PCT, PSTD, PJC, or PRC), continuous ensemble statistics (ECNT), observation rank statistics (ORANK) or ranked probability score (RPS) line types in the **output_flag** dictionary. The number of probabilistic HiRA output lines is determined by the number of categorical forecast thresholds and HiRA neighborhood widths chosen.
 
 The HiRA framework provides a unique method for evaluating models in the neighborhood of point observations, allowing for some spatial and temporal uncertainty in the forecast and/or the observations. Additionally, the HiRA framework can be used to compare deterministic forecasts to ensemble forecasts. In MET, the neighborhood is a circle or square centered on the grid point closest to the observation location. An event is defined, then the proportion of points with events in the neighborhood is calculated. This proportion is treated as an ensemble probability, though it is likely to be uncalibrated.
 
@@ -425,6 +425,7 @@ ________________________
      pjc    = BOTH;
      prc    = BOTH;
      ecnt   = BOTH;  // Only for HiRA
+     orank  = BOTH;  // Only for HiRA
      rps    = BOTH;  // Only for HiRA
      eclv   = BOTH;
      mpr    = BOTH;
@@ -450,9 +451,9 @@ The **output_flag** array controls the type of output that the Point-Stat tool g
 
 9. **VL1L2** for Vector L1L2 Partial Sums
 
-10. **VCNT** for Vector Continuous Statistics (Note that bootstrap confidence intervals are not currently calculated for this line type.)
+10. **VAL1L2** for Vector Anomaly L1L2 Partial Sums when climatological data is supplied
 
-11. **VAL1L2** for Vector Anomaly L1L2 Partial Sums when climatological data is supplied
+11. **VCNT** for Vector Continuous Statistics
 
 12. **PCT** for Contingency Table counts for Probabilistic forecasts
 
@@ -464,13 +465,15 @@ The **output_flag** array controls the type of output that the Point-Stat tool g
 
 16. **ECNT** for Ensemble Continuous Statistics is only computed for the HiRA methodology
 
-17. **RPS** for Ranked Probability Score is only computed for the HiRA methodology
+17. **ORANK** for Ensemble Matched Pair Information when point observations are supplied for the HiRA methodology
 
-18. **ECLV** for Economic Cost/Loss Relative Value
+18. **RPS** for Ranked Probability Score is only computed for the HiRA methodology
 
-19. **MPR** for Matched Pair data
+19. **ECLV** for Economic Cost/Loss Relative Value
 
-Note that the first two line types are easily derived from each other. Users are free to choose which measures are most desired. The output line types are described in more detail in :numref:`point_stat-output`.
+20. **MPR** for Matched Pair data
+
+Note that the FHO and CTC line types are easily derived from each other. Users are free to choose which measures are most desired. The output line types are described in more detail in :numref:`point_stat-output`.
 
 Note that writing out matched pair data (MPR lines) for a large number of cases is generally not recommended. The MPR lines create very large output files and are only intended for use on a small set of cases.
 
@@ -489,9 +492,9 @@ point_stat_PREFIX_HHMMSSL_YYYYMMDD_HHMMSSV.stat where PREFIX indicates the user-
 
 The output ASCII files are named similarly:
 
-point_stat_PREFIX_HHMMSSL_YYYYMMDD_HHMMSSV_TYPE.txt where TYPE is one of mpr, fho, ctc, cts, cnt, mctc, mcts, pct, pstd, pjc, prc, ecnt, rps, eclv, sl1l2, sal1l2, vl1l2, vcnt or val1l2 to indicate the line type it contains.
+point_stat_PREFIX_HHMMSSL_YYYYMMDD_HHMMSSV_TYPE.txt where TYPE is one of mpr, fho, ctc, cts, cnt, mctc, mcts, pct, pstd, pjc, prc, ecnt, orank, rps, eclv, sl1l2, sal1l2, vl1l2, vcnt or val1l2 to indicate the line type it contains.
 
-The first set of header columns are common to all of the output files generated by the Point-Stat tool. Tables describing the contents of the header columns and the contents of the additional columns for each line type are listed in the following tables. The ECNT line type is described in :numref:`table_ES_header_info_es_out_ECNT`. The RPS line type is described in :numref:`table_ES_header_info_es_out_RPS`.
+The first set of header columns are common to all of the output files generated by the Point-Stat tool. Tables describing the contents of the header columns and the contents of the additional columns for each line type are listed in the following tables. The ECNT line type is described in :numref:`table_ES_header_info_es_out_ECNT`. The ORANK line type is described in :numref:`table_ES_header_info_es_out_ORANK`. The RPS line type is described in :numref:`table_ES_header_info_es_out_RPS`.
 
 .. _table_PS_header_info_point-stat_out:
 

met/scripts/config/PointStatConfig

@@ -199,6 +199,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE; // Only for HiRA
+   orank  = NONE; // Only for HiRA
    rps    = NONE; // Only for HiRA
    eclv   = BOTH;
    mpr    = BOTH;

met/src/tools/core/point_stat/point_stat.cc

@@ -327,7 +327,9 @@ void setup_first_pass(const DataPlane &dp, const Grid &data_grid) {
 ////////////////////////////////////////////////////////////////////////
 
 void setup_txt_files() {
-   int i, max_col, max_prob_col, max_mctc_col, n_prob, n_cat, n_eclv;
+   int i, j;
+   int max_col, max_prob_col, max_mctc_col, max_orank_col;
+   int n_prob, n_cat, n_eclv, n_ens;
    ConcatString base_name;
 
    // Create output file names for the stat file and optional text files
@@ -340,23 +342,20 @@ void setup_txt_files() {
    /////////////////////////////////////////////////////////////////////
 
    // Get the maximum number of data columns
-   n_prob = conf_info.get_max_n_fprob_thresh();
+   n_prob = max(conf_info.get_max_n_fprob_thresh(),
+                conf_info.get_max_n_hira_prob());
    n_cat  = conf_info.get_max_n_cat_thresh() + 1;
    n_eclv = conf_info.get_max_n_eclv_points();
+   n_ens  = conf_info.get_max_n_hira_ens();
 
-   // Check for HiRA output
-   for(i=0; i<conf_info.get_n_vx(); i++) {
-      if(conf_info.vx_opt[i].hira_info.flag) {
-         n_prob = max(n_prob, conf_info.vx_opt[i].hira_info.cov_ta.n());
-      }
-   }
-
-   max_prob_col = get_n_pjc_columns(n_prob);
-   max_mctc_col = get_n_mctc_columns(n_cat);
+   max_prob_col  = get_n_pjc_columns(n_prob);
+   max_mctc_col  = get_n_mctc_columns(n_cat);
+   max_orank_col = get_n_orank_columns(n_ens);
 
    // Determine the maximum number of data columns
-   max_col = ( max_prob_col > max_stat_col ? max_prob_col : max_stat_col );
-   max_col = ( max_mctc_col > max_col      ? max_mctc_col : max_col );
+   max_col = (max_prob_col  > max_stat_col ? max_prob_col  : max_stat_col);
+   max_col = (max_mctc_col  > max_col      ? max_mctc_col  : max_col);
+   max_col = (max_orank_col > max_col      ? max_orank_col : max_col);
 
    // Add the header columns
    max_col += n_header_columns + 1;
@@ -429,6 +428,10 @@ void setup_txt_files() {
                max_col = get_n_eclv_columns(n_eclv) + n_header_columns + 1;
                break;
 
+            case(i_orank):
+               max_col = get_n_orank_columns(n_ens) + n_header_columns + 1;
+               break;
+
             default:
                max_col = n_txt_columns[i] + n_header_columns + 1;
                break;
@@ -465,6 +468,10 @@ void setup_txt_files() {
                write_eclv_header_row(1, n_eclv, txt_at[i], 0, 0);
                break;
 
+            case(i_orank):
+               write_orank_header_row(1, n_ens, txt_at[i], 0, 0);
+               break;
+
             default:
                write_header_row(txt_columns[i], n_txt_columns[i], 1,
                   txt_at[i], 0, 0);
@@ -1749,6 +1756,18 @@ void do_hira_ens(int i_vx, const PairDataPoint *pd_ptr) {
             txt_at[i_ecnt], i_txt_row[i_ecnt]);
       } // end if ECNT
 
+      // Write out the ORANK line
+      if(conf_info.vx_opt[i_vx].output_flag[i_orank] != STATOutputType_None) {
+
+         // Compute ensemble statistics
+         hira_pd.compute_pair_vals(rng_ptr);
+
+         write_orank_row(shc, &hira_pd,
+            conf_info.vx_opt[i_vx].output_flag[i_orank],
+	         stat_at, i_stat_row,
+	         txt_at[i_orank], i_txt_row[i_orank]);
+      } // end if ORANK
+
       // Write out the RPS line
       if(conf_info.vx_opt[i_vx].output_flag[i_rps] != STATOutputType_None) {
 

met/src/tools/core/point_stat/point_stat.h

@@ -70,8 +70,8 @@ static const char **txt_columns[n_txt] = {
    sl1l2_columns,  sal1l2_columns, vl1l2_columns,
    val1l2_columns, pct_columns,    pstd_columns,
    pjc_columns,    prc_columns,    ecnt_columns,
-   rps_columns,    eclv_columns,   mpr_columns,
-   vcnt_columns
+   orank_columns,  rps_columns,    eclv_columns,
+   mpr_columns,    vcnt_columns
 };
 
 // Length of header columns
@@ -81,8 +81,8 @@ static const int n_txt_columns[n_txt] = {
    n_sl1l2_columns,  n_sal1l2_columns, n_vl1l2_columns,
    n_val1l2_columns, n_pct_columns,    n_pstd_columns,
    n_pjc_columns,    n_prc_columns,    n_ecnt_columns,
-   n_rps_columns,    n_eclv_columns,   n_mpr_columns,
-   n_vcnt_columns
+   n_orank_columns,  n_rps_columns,    n_eclv_columns,
+   n_mpr_columns,    n_vcnt_columns
 };
 
 // Text file abbreviations
@@ -92,8 +92,8 @@ static const char *txt_file_abbr[n_txt] = {
    "sl1l2",  "sal1l2", "vl1l2",
    "val1l2", "pct",    "pstd",
    "pjc",    "prc",    "ecnt",
-   "rps",    "eclv",   "mpr",
-   "vcnt"
+   "orank",  "rps",    "eclv",
+   "mpr",    "vcnt"
 };
 
 ////////////////////////////////////////////////////////////////////////

met/src/tools/core/point_stat/point_stat_conf_info.cc

@@ -523,6 +523,26 @@ int PointStatConfInfo::get_max_n_eclv_points() const {
 
 ////////////////////////////////////////////////////////////////////////
 
+int PointStatConfInfo::get_max_n_hira_ens() const {
+   int i, n;
+
+   for(i=0,n=0; i<n_vx; i++) n = max(n, vx_opt[i].get_n_hira_ens());
+
+   return(n);
+}
+
+////////////////////////////////////////////////////////////////////////
+
+int PointStatConfInfo::get_max_n_hira_prob() const {
+   int i, n;
+
+   for(i=0,n=0; i<n_vx; i++) n = max(n, vx_opt[i].get_n_hira_prob());
+
+   return(n);
+}
+
+////////////////////////////////////////////////////////////////////////
+
 bool PointStatConfInfo::get_vflag() const {
    int i;
    bool vflag = false;
@@ -1194,7 +1214,7 @@ int PointStatVxOpt::n_txt_row(int i_txt_row) const {
 
       case(i_ecnt):
       case(i_rps):
-         // Number of HiRA ECNT and RPS lines =
+         // Number of HiRA ECNT, ORANK and RPS lines =
          //    Message Types * Masks * Interpolations * HiRA widths *
          //    Alphas
          if(hira_info.flag) {
@@ -1206,6 +1226,20 @@ int PointStatVxOpt::n_txt_row(int i_txt_row) const {
 
          break;
 
+      case(i_orank):
+         // Number HiRA ORANK lines possible =
+         //    Number of pairs * Categorical Thresholds *
+         //    HiRA widths
+         if(hira_info.flag) {
+            n = vx_pd.get_n_pair() * get_n_cat_thresh() *
+                hira_info.width.n();
+         }
+         else {
+            n = 0;
+         }
+
+         break;
+
       case(i_eclv):
          // Number of CTC -> ECLV lines =
          //    Message Types * Masks * Interpolations * Thresholds *
@@ -1282,3 +1316,16 @@ int PointStatVxOpt::get_n_oprob_thresh() const {
 }
 
 ////////////////////////////////////////////////////////////////////////
+
+int PointStatVxOpt::get_n_hira_ens() const {
+   int n = (hira_info.flag ? hira_info.width.max() : 0);
+   return(n*n);
+}
+
+////////////////////////////////////////////////////////////////////////
+
+int PointStatVxOpt::get_n_hira_prob() const {
+   return(hira_info.flag ? hira_info.cov_ta.n() : 0);
+}
+
+////////////////////////////////////////////////////////////////////////

met/src/tools/core/point_stat/point_stat_conf_info.h

@@ -46,12 +46,13 @@ static const int i_pjc       = 12;
 static const int i_prc       = 13;
 static const int i_ecnt      = 14;
 
-static const int i_rps       = 15;
-static const int i_eclv      = 16;
-static const int i_mpr       = 17;
-static const int i_vcnt      = 18;
+static const int i_orank     = 15;
+static const int i_rps       = 16;
+static const int i_eclv      = 17;
+static const int i_mpr       = 18;
+static const int i_vcnt      = 19;
 
-static const int n_txt       = 19;
+static const int n_txt       = 20;
 
 // Text file type
 static const STATLineType txt_file_type[n_txt] = {
@@ -74,10 +75,11 @@ static const STATLineType txt_file_type[n_txt] = {
    stat_prc,    //  13
    stat_ecnt,   //  14
 
-   stat_rps,    //  14
-   stat_eclv,   //  15
-   stat_mpr,    //  16
-   stat_vcnt,   //  17
+   stat_orank,  //  15
+   stat_rps,    //  16
+   stat_eclv,   //  17
+   stat_mpr,    //  18
+   stat_vcnt,   //  19
 
 };
 
@@ -178,19 +180,21 @@ class PointStatVxOpt {
       int get_n_oprob_thresh() const;
 
       int get_n_eclv_points()  const;
+      int get_n_hira_ens()     const;
+      int get_n_hira_prob()    const;
       int get_n_cdf_bin()      const;
       int get_n_ci_alpha()     const;
 };
 
 ////////////////////////////////////////////////////////////////////////
 
-inline int PointStatVxOpt::get_n_msg_typ()     const { return(msg_typ.n_elements());         }
-inline int PointStatVxOpt::get_n_mask()        const { return(mask_name.n_elements());       }
-inline int PointStatVxOpt::get_n_interp()      const { return(interp_info.n_interp);         }
+inline int PointStatVxOpt::get_n_msg_typ()     const { return(msg_typ.n());          }
+inline int PointStatVxOpt::get_n_mask()        const { return(mask_name.n());        }
+inline int PointStatVxOpt::get_n_interp()      const { return(interp_info.n_interp); }
 
-inline int PointStatVxOpt::get_n_eclv_points() const { return(eclv_points.n_elements());     }
-inline int PointStatVxOpt::get_n_cdf_bin()     const { return(cdf_info.n_bin);               }
-inline int PointStatVxOpt::get_n_ci_alpha()    const { return(ci_alpha.n_elements());        }
+inline int PointStatVxOpt::get_n_eclv_points() const { return(eclv_points.n());      }
+inline int PointStatVxOpt::get_n_cdf_bin()     const { return(cdf_info.n_bin);       }
+inline int PointStatVxOpt::get_n_ci_alpha()    const { return(ci_alpha.n());         }
 
 ////////////////////////////////////////////////////////////////////////
 
@@ -266,6 +270,8 @@ class PointStatConfInfo {
       int get_max_n_fprob_thresh() const;
       int get_max_n_oprob_thresh() const;
       int get_max_n_eclv_points()  const;
+      int get_max_n_hira_ens()     const;
+      int get_max_n_hira_prob()    const;
 
       // Check for any verification of vectors
       bool get_vflag() const;

test/config/PointStatConfig_APCP

@@ -120,6 +120,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE;
+   orank  = NONE;
    rps    = NONE;
    eclv   = BOTH;
    mpr    = NONE;

test/config/PointStatConfig_APCP_HIRA

@@ -123,6 +123,7 @@ output_flag = {
    pjc    = STAT;
    prc    = STAT;
    ecnt   = STAT;
+   orank  = STAT;
    rps    = STAT;
    eclv   = STAT;
    mpr    = STAT;

test/config/PointStatConfig_GTG_latlon

@@ -142,6 +142,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE;
+   orank  = NONE;
    rps    = NONE;
    eclv   = BOTH;
    mpr    = BOTH;

test/config/PointStatConfig_GTG_lc

@@ -150,6 +150,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE;
+   orank  = NONE;
    rps    = NONE;
    eclv   = BOTH;
    mpr    = BOTH;

test/config/PointStatConfig_INTERP_OPTS

@@ -133,6 +133,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE;
+   orank  = NONE;
    rps    = NONE;
    eclv   = NONE;
    mpr    = STAT;

test/config/PointStatConfig_LAND_TOPO_MASK

@@ -173,6 +173,7 @@ output_flag = {
    pjc    = NONE;
    prc    = NONE;
    ecnt   = NONE;
+   orank  = NONE;
    rps    = NONE;
    eclv   = NONE;
    mpr    = BOTH;