Incorrect thresholds printed from stat_analysis

[johnlwagner]

Greetings
We have had a handful of cases now where the FCST_THRESH printed in the stat output file from stat_analysis is incorrect. We're running stat_analysis on WCOSS (venus in this case) using MET V10.0 from /gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles. This problem has occurred when running stat_analysis to aggregate station scores for elements with multiple thresholds (qpf06 and sky cover). In the qpf06 example below, the top FCST_THRESH printed is >=6.35, when >=0.254 should have been printed, as it was for the 2 lines below it.
Rerunning this case seems to solve the issue, so it doesn't seem to be an issue with stat_analysis itself (at least its not a repeatable problem). I'm wondering if the issue is related to the resources given to stat_analysis. We run in the dev2 queue, typically with 40 tasks running per node.
Has anyone else running on WCOSS run into similar issues? If so, do you have any recommendations for setting the resources on a node? Have you run with fewer than 40 tasks in order to make more memory available for each task? Any recommendations would be appreciated.
Thanks
John

[j-opatz]

Hi John,

And thank you for your question. Unfortunately, an issue like the one you've indicated can be very elusive: an unrepeatable problem is a problem that needs to be tracked down. While this issue may have a codified solution in MET, the time and cost of finding out exactly what it is is prohibitive to taking any steps at the moment. It's very possible that it is being caused by the taskload, but we can't say it with certainty.

If this problem becomes a consistent, repeatable issue we will take a look at the environment and run commands and try to find a solution. In the meantime, I would suggest that if you find a lower task number doesn't cause these issues to continue at that lower task load.

[JohnHalleyGotway]

@johnlwagner I did talk with @j-opatz about this topic and am obviously concerned. I’ve been thinking about it for a few days but haven’t come up with a great way of approaching the problem yet.

Some potential problems that might manifest themselves in this way are uninitialized memory and array overflow. But I don’t have great confidence in that “diagnosis”.

It’s also possible that MET 10.1.0 will fix this behavior, released yesterday. But that may also just be wishful thinking.

At this point, I’d like to grab some of the data you’re using as input to Stat-Analysis along with the job or config file you’re running. Can you please point me to those details on WCOSS?

I can recompile MET locally with some flags to tell it to initialize memory to weird values. That sometimes helps make memory issues more repeatable.

[JohnHalleyGotway]

@johnlwagner we haven't had any activity on this discussion in the last 2 weeks. Wondering if you're still seeing this behavior on WCOSS? If you'd like me to go sift through data on WCOSS to look for anything obvious, please point me to their location.

[johnlwagner]

Sorry for the delay on this. We continue to reprocess our data, but we have not been able to repeat the error yet. I did set up a test case to show how we are running stat_analysis. On venus (WCOSS), please look in /gpfs/dell3/mdl/mdlverif/noscrub/usr/John.L.Wagner/mdl-verification/MET_test_case. In there, you will find a test script, a month's worth of data, the stat_analysis config file, and the output of stat_analysis. If you need any additional information, please let me know. Thanks John

[JohnHalleyGotway]

@johnlwagner I did want to mention something about this setup for Stat-Analysis.

1. In run_stat_analysis.sh, change...
   FROM: export fcstList=`ls /gpfs/dell3/mdl/mdlverif/noscrub/usr/John.L.Wagner/mdl-verification/MET_test_case/*.stat```` TO: export fcstList=`ls /gpfs/dell3/mdl/mdlverif/noscrub/usr/John.L.Wagner/mdl-verification/MET_test_case/point_stat*.stat```

Just using *.stat includes any .stat files previously generated by running previous calls to Stat-Analysis in that directory. Might be safer to stash input and output in separate directories. But at a minimum, using point_stat*.stat would prevent you from recycling old output into new Stat-Analysis calls.

Increasing the verbosity level to 3 -v 3 confirms that Stat-Analysis is reading the output it previously generated:

DEBUG 3: Processing STAT file "/u/John.H.Gotway/MET_test_case/validtime_point_ndfd20210118_12_masknum1_cnt.stat" .... 31 of 35
DEBUG 3: Processing STAT file "/u/John.H.Gotway/MET_test_case/validtime_point_ndfd20210118_12_masknum1_ctc.stat" .... 32 of 35
DEBUG 3: Processing STAT file "/u/John.H.Gotway/MET_test_case/validtime_point_ndfd20210118_12_masknum1_cts.stat" .... 33 of 35
DEBUG 3: Processing STAT file "/u/John.H.Gotway/MET_test_case/validtime_point_ndfd20210118_12_masknum1_mctc.stat" ... 34 of 35
DEBUG 3: Processing STAT file "/u/John.H.Gotway/MET_test_case/validtime_point_ndfd20210118_12_masknum1_mcts.stat" ... 35 of 35

2. I also see lots of warnings like this:

WARNING: Dictionary::lookup_string() -> string lookup failed for name "fcst_valid_beg"

Checking the run script, I'm not sure why. I see this:

run_stat_analysis.sh:export FCST_VALID_BEGIN=20220101
stat_analysis_config_point_cl:fcst_valid_beg  = ["${FCST_VALID_BEGIN}"];

Then I realized that several of these options are NOT arrays, as you have them set in the config file.
Changing stat_analysis_config_point_cl...
FROM: fcst_valid_beg = ["${FCST_VALID_BEGIN}"];
TO: fcst_valid_beg = "${FCST_VALID_BEGIN}";
Makes those warnings go away.

Also need to change...
FROM:vx_mask = { field =["${MASKS}"]; };
TO:vx_mask = ["${MASKS}"];

And just remove the "fcst_thresh" and "obs_thresh" settings. Not sure what you're trying to accomplish there.

But with all those changes, we end up with 0 lines being written to the output. I think that by making the filtering settings actually work, we're filtering out all the input from the point_stat*.stat files.

If you'd like, we could schedule a meeting to step through your Stat-Analysis config file and discuss the filtering logic for each of the jobs.

[johnlwagner]

Thanks John. I would appreciate your help in going over our stat-analysis config files. Let me know what days/times work for you. My calendar is mostly open Thursday-Friday this week and Tuesday-Friday next week. The first issue that you reported below I don't foresee being an issue in our normal processing. We copy the data to a run-specific directory in ptmp on WCOSS, so there would not be previous runs that we need to worry about. We will start testing the rest of your suggestions this week.

[JohnHalleyGotway]

@johnlwagner we met last week to review MDL's use of MET. I look forward to hearing more about it at the upcoming METplus workshop. Wondering if we can mark this discussion as being answered since we've done additional followup in the meeting and emails?

[johnlwagner]

Yes, we can mark this as answered. Thanks for your help!