Add LEPS and SEEPS to MET probabilistic verification.

[dwfncar]

Replace italics below with details for this issue.

Describe the New Feature

On 10/29/2015, Geoff Dimego at NCEP inquired as to whether MET can compute SEEPS. Tara responded that no, it can't, but it could be added in the next release if we can find funding. Here's some more details on it...

According to Beth Ebert's website (http://www.cawcr.gov.au/projects/verification/) both LEPS and SEEPS are computed relative to climatology. And they require knowledge of the climatological distribution. In met-5.1, we read the climatological mean but not the spread. Adding LEPS and SEEPS would require us to read the spread. If I understand correctly, we convert the actual observed event to the likelihood that value would occur based on the climatological PDF (likely assuming a normal distribution). And we use that observed likelihood to compute these scores.

I suppose we would add LEPS and SEEPS to the PSTD line type... although they can't be derived from the Nx2 probabilistic contingency table.

Also, if these require the climatological spread, we'd need to add that to the MPR line type so that SEEPS and LEPS could be computed from the MPR line type. And if we add the climo spread to MPR, we should add it to ORANK to keep things consistent.

Marion Mittermaier says Rachael North can help with development and testing. She also pointed out that Rachael now has a method for SEEPS using TRMM so we will need to implement in Grid Stat as well. Finally, Marion mentioned that LEPS is not really related to SEEPS. It's a continuous statistics.

Stat-Analysis will also need to be modified to add this into the WMO/CBS format

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

MET-10.0

Funding Source

2799991 - Met Office

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required RACHAEL NORTH

Labels

• Select component(s)
• Select priority
• [ X Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

On 10/29/2015, Geoff Dimego at NCEP inquired as to whether MET can compute SEEPS. Tara responded that no, it can't, but it could be added in the next release if we can find funding. Here's some more details on it...

According to Beth Ebert's website (http://www.cawcr.gov.au/projects/verification/) both LEPS and SEEPS are computed relative to climatology. And they require knowledge of the climatological distribution. In met-5.1, we read the climatological mean but not the spread. Adding LEPS and SEEPS would require us to read the spread. If I understand correctly, we convert the actual observed event to the likelihood that value would occur based on the climatological PDF (likely assuming a normal distribution). And we use that observed likelihood to compute these scores.

I suppose we would add LEPS and SEEPS to the PSTD line type... although they can't be derived from the Nx2 probabilistic contingency table.

Also, if these require the climatological spread, we'd need to add that to the MPR line type so that SEEPS and LEPS could be computed from the MPR line type. And if we add the climo spread to MPR, we should add it to ORANK to keep things consistent. [MET-563] created by johnhg

[dwfncar]

Ying Lin is using this so we will need to add either in 6.0 or 6.0plus
by jensen

[dwfncar]

Made critical because it adds columns to output and breaks METViewer
by jensen

[dwfncar]

Charge USWRP - 775052
by jensen

[dwfncar]

SEEPS requires gauge point QPF climatology data. See https://sdg/jira/browse/@PSTARTGH-846@PEND
by johnhg

[dwfncar]

On Tue, Aug 21, 2018 at 8:24 AM, Martin Janousek <martin.janousek@ecmwf.int> wrote:

    Tara,

    I am sending you the information on the SEEPS computation and supplementary data.

    SEEPS (Stable Equitable Error in Probability Space) is a score which measures quality of the precipitation (deterministic) forecasts. It is based on special climatology data derived from precipitation observations which based on the cumulative distribution function determines at each station the thresholds between dry, light and heavy precipitation categories, specific for each station.

    References:
    - Rodwell, M. J., D. S. Richardson, T. D. Hewson, and T. Haiden, 2010: A new equitable score suitable for verifying precipitation in numerical weather prediction. Q. J. R. Meteorol. Soc., 136, 1344-1363.
    - Somewhat more concise info in the ECMWF Technical Memorandum 665 (https://www.ecmwf.int/sites/default/files/elibrary/2012/9732-intercomparison-global-model-precipitation-forecast-skill-201011-using-seeps-score.pdf)
    Please note the section "2.4 Areal averaging and aggregation" in TM665 which explains how we compute spatial weights for stations based on their density when computing areal means of SEEPS (one of the actions on ECMWF we identified during our visit to Boulder last year).

    The computation of the SEEPS consists of determining in which cell of 3x3 table of the climatology the precipitation forecast and observation lies and then using the value from the cell as the SEEPS value at that station; then summing-up over the area and/or the period. So the computation is simple providing one has the SEEPS climatology available. I attach the link to the current SEEPS climatology we use at ECMWF; this is the same climatology ECMWF provides to the global centres participating in the exchange of verification scores under the WMO coordination. The archive file contains also the description how the climatology was build in case you wish e.g. to create your own climatology based on better observation set over U.S. And it also contains a simple C-function illustrating how SEEPS is computed.

    The climatology dataset can be accessed from https://drive.google.com/file/d/1j6Uszjl7Cetn0cKxVFr747tldKRvqjG_ (13MB).

    Please let me know if you need more information.

    Cheers,

    Martin by jensen

[dwfncar]

This request was reiterated during the NCWCP METplus tutorial in October 2018.

The crux here is adding support for point-based climatologies. by johnhg

[TaraJensen]

Charge 2780541

[JohnHalleyGotway]

Closing this issue since it is superseded by #1941, #1942, and #1943.