Enhance TC-Pairs to include storm diagnostics in consensus track output

[JohnHalleyGotway]

Describe the Enhancement

When the -diag command line option is provided, the TC-Pairs tool writes TCDIAG output lines after the TCMPR lines to which they correspond. However these diagnostics are NOT currently included in the computation of consensus tracks.

This issue is to enhance the computation of consensus tracks to include the average of the model diagnostics of the members. There are obvious questions about whether or not each diagnostic must be present and valid in the members for it to be included in the consensus output. Does this need to be configurable?

Time Estimate

2 days?

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2770043

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  None.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

@sethlinden and @JohnHalleyGotway met on 7/31/23 to discuss.

1. What's a good test? Let's add a new one to unit_tc_pairs.xml:
   Recommend coordinating with @jvigh to get an ensemble of LSDIAG data.

For development purposes, you can just copy/modify this sample LSDIAG file:
$MET_TEST_INPUT/tc_data/diag/ships_diag_rt/2022/22092818AL0922_lsdiag.dat

The TC-Pairs test for this only needs to compute a single consensus track containing at least 2 members.

2. Overview of code changes needed:

Probably do NOT need to add a consensus.compute_diag flag. If you don't want consensus diagnostics, just don't use the -diag command line option to provide them.

The TrackInfo::DiagName string array stores the names of the diagnostics extracted from the input files. The TrackPoint::DiagVal numeric array stores the values for those diagnostics for that track point. They will always have the same length.

Enhance the consensus function in track_info.cc to handle these diagnostics.

• Loop through the consensus members and compute the union of the diagnostic names as a StringArray.

for(i=0; i<input track; i++) {
  for(j=0;j<track[i].diag_name.n(); j++) {
     diag_names.add_unique(diag_name[j]);
  }
}

• For each unique diagnostic name and track point, compute the mean of the diagnostic across the members.
• Consider adding TrackInfo::get_diag_val(int track_point_index, const char *diag_name) as a convenience function to retrieve diagnostic values for each track point by name. For each diagnostic name, store the diag values in a NumArray object and compute the consensus value as NumArray::mean(). Check for bad data before storing in the NumArray.

track_info.cc.txt

[JohnHalleyGotway]

@sethlinden please find this new tc_pairs_DIAGNOSTICS_CONSENSUS unit test available on the feature_2476_tc_pair_diag branch to demonstrate calling TC-Pairs to compute a consensus track from 2 input tracks that have diagnostics attached to them.

Currently, the al092022_20220926_DIAGNOSTICS_CONSENSUS.tcst output file contains tracks for the GFSI and EMXI ATCF ID's WITH TCDIAG lines present, and tracks for the GFEX_CONS consensus WITHOUT TCDIAG lines present.

The task for this issue is to enhance TC-Pairs and/or MET library code to include the TCDIAG lines for the GFEX_CONS consensus model output.

Please note some important details.

1. The input data for this test IS available on seneca in this EMXI directory, GFSI directory, and complete aal092022.dat ADECK file.
2. This EMXI and GFSI CIRA diagnostics data is totally fake. It is literally a copy of these AVNO diag files but I changed the file name to emxi or gfsi and updated the ATCF ID inside the files in the same way.
3. Since they're both copies of the same input, the diagnostics will be exactly the same. And therefore, the consensus diagnostics will also be the same. You could consider modifying some of the values in those files to confirm that the mean is actually working.
4. This fake data facilitates development for now. However, I'd recommend that @jvigh and/or @KathrynNewman provide more realistic sample data instead. When they do, please discard this fake data in favor of what they provide.
5. Note that while I did add these files to this directory (https://dtcenter.ucar.edu/dfiles/code/METplus/MET/MET_unit_test/unit_test) I DID NOT recreate this tarfile:
   https://dtcenter.ucar.edu/dfiles/code/METplus/MET/MET_unit_test/develop/unit_test-all.tgz
   So the new test will run on seneca but WILL NOT run via GitHub actions. That would require recreating that tarfile.
6. Please also review the updated description of the match_to_track config file option in this version of the MET User's Guide. The key change is clarifying what happens when match_to_track = [ ]; for CIRA_DIAG_RT inputs.

[jvigh]

@sethlinden and @JohnHalleyGotway - based on our brief discussion in today's MET Engineering meeting, I prepared the following for the agenda for next week's TCDiag project-wide meeting. Please review this and let me know if there are any other TCDiag-consensus issues that you need guidance on. Sorry that the intro is a bit incomplete -- I forgot what other steps beside interpolation are accomplished by the code. You might add a link to the relevant section of code where these are occurring.

Discuss whether the diagnostics should be interpolated before a consensus is computed. Seth Linden is working on MET #2476. In general, the consensus code for track and intensity performs a number of processing steps, such as interpolating the model data and other steps. However, it is not known what is desired for the TCDiag output. Questions to discuss:

• Should TCDiag be interpolated in time to bring it up to the level of an “early” model, like how the SHIPS diagnostics are presented?
• Should this happen before a consensus is computed?
• In general, should the TCDiagnostics be using 6-h old fields to compute early diagnostics?
• What other processing should occur before the consensus of the diagnostic values are computed?

[KathrynNewman]

@jvigh - just to clarify, the interp12 option in TC-Pairs is not the same as running the interpolator. The functionality is to rename previously interpolated forecasts. To get get early guidance, you'd need to run the interpolator code - which isn't in METplus. I might be missing the full context here, but I wanted to jump in that TC-Pairs doesn't currently perform interpolation as it's done at NHC.

[JohnHalleyGotway]

Based on feedback from @musgrave-kate on 9/20/23, recommend adding the following:

1. Print a warning if diagnostics are present for AT LEAST one member but not present for ALL members.
2. Add two new config options in the consensus dictionary entries to provide finer handling of diagnostics:

• diag_min_req is similar to the min_req entry to specify the required number of diagnostics present to be included in the output.
• diag_required is similar to the required entry to specify the models that must contain diagnostics for them to be calculated.

[jvigh]

@sethlinden Have you gotten ensemble diagnostics data from Kate Musgrave?

[JohnHalleyGotway]

@jvigh we didn't get the ensemble track data from @musgrave-kate yet, but I did note the need for that change in this follow on #2699 issue.

[JohnHalleyGotway]

Reopening this issue since I was too hasty in approving/merging PR #2694.
This GHA run actually did flag an unexpected difference:

COMPARING tc_pairs/al132020_CONSENSUS.tcst
file1: /data/output/met_test_truth/tc_pairs/al132020_CONSENSUS.tcst
file2: /data/output/met_test_output/tc_pairs/al132020_CONSENSUS.tcst
ERROR: differing number of rows 1079 vs. 1028 for row type TCST_TCMPR between versions 12_0 vs. 12_0 

Inspecting the failed NB output on seneca, I see that the difference is caused by 51 lines for the UEMN_CONS consensus no longer being present in the new output:

cd /d1/projects/MET/MET_regression/develop/NB20231005
for model in `cat MET-develop-ref/test_output/tc_pairs/al132020_CONSENSUS.tcst MET-develop/test_output/tc_pairs/al132020_CONSENSUS.tcst | awk '{print $2}' | sort -u | egrep -v AMODEL`; do
  echo $model `grep " $model " MET-develop-ref/test_output/tc_pairs/al132020_CONSENSUS.tcst | wc -l` `grep " $model " MET-develop/test_output/tc_pairs/al132020_CONSENSUS.tcst | wc -l`
done
...
UEMN_CONS 51 0

This change is unexpected, and I'll investigate.

• I note that UEMN_CONS is first consensus defined in TCPairsConfig_CONSENSUS.
• Rerunning at debug level 4 prints these telling log messages:

38058 DEBUG 2: Added 4 CLIPER/SHIFOR baseline track(s).
38059 DEBUG 2: Filtering 210 ADECK tracks based on config file settings.
38060 DEBUG 4: Discarding track 15 since it is listed in SkipConsensusMembers: UE00
38061 DEBUG 4: Discarding track 16 since it is listed in SkipConsensusMembers: UE01
...
38250 DEBUG 4: [Case 1] For case "AL 13 20200823_000000" skipping consensus model "UEMN_CONS" since the minimum number of required members were not found (0 < 36).'

• TC-Pairs is discarding the member track PRIOR TO using them to compute the consensus.
• The fix is pretty easy, just moving the call to filter_tracks(...) down immediately after the call to derive_consensus(...).