Update develop-ref after #1754 #1758
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
January 26, 2021 09:32
* Try path insert. * sys.path insert. * Per #1319, adding David's changes back into the feature_1319_no_pickle branch. It compiles but TEST: python_numpy_plot_data_plane_pickle fails when testing on my Mac. Comitting now to test on kiowa. * Per #1319, small updated to write_tmp_dataplane.py script. Had a couple of if statements that should really be elif. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Per #1736, if -out_stat was used for aggregate or aggregate_stat jobs, do not write output to the -out or log output. * Per #1736, clarify stat_analysis documentation for -out_stat option. * Per #1736, for jobs which can write .stat output, don't waste time populating the output AsciiTable unless it's actually going to be written. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
…_python.xml works via the command line, it fails when run through cron. The problem is the PATH setting. Need to have the anaconda bin directory in the path for it to succeeed. Adding that for the single test.
…les set prior to the test are unset afterwards! So we'd run all the rest of the tests after unit_python.xml with an empty path. That would likely cause any subsequent call to Rscript to fail. Recommend tightening up this logic when we move these tests to GHA.
…vailable for python embedding cases that use MET_PYTHON_EXE
* Per #1747, update MET to interpret longlong values as integers. NetCDF file attributes that have an LL suffix are read into python as numpy.int64 objects. Right now MET fails when trying to read those as integers. Update the parsing logic to interpret those as ints. * Per #1747, since MET can now interpret both long and longlong's as ints, there's no need to cast nx and ny to ints in the read_tmp_dataplane.py script anymore. * Per #1747, this is slightly unrelated. But after installing the netCDF4 module on kiowa for /usr/local/met-python3/bin/python3, we should no longer need a custom PATH setting to get unit_python.xml to work. Reverting the change I made to it a couple of days ago to get it working.
…kept failing through GHA with a divide by zero error. It occurs in compute_track_err() but only for a very specific set of data. The bdeck valid increment evaluates to 0 which causes the divide by 0 error. It also can evaluate to bad data (e.g. -9999). The fix is to check for 0 and bad data. If found, use the constant best_track_time_step value instead.
…numbers can change
* Per #1714, add tc_gen genesis_match_window configuration option to define a search window relative to the forecast genesis time. * Per #1714, clarify docs to state the genesis_match_window.end = 12 allows for matches for early forecasts. Also add an example of this option to the tc_gen unit test. * Per #1714, switch ops_hit_tdiff to ops_hit_window. * Per #1714, skip genesis events for tracks where the cyclone number is > 50. * Per #1714, only discard cyclone numbers > 50 from the Best track, not the forecast tracks. * Per #1716, add note to the tc_gen chapter about skipping Best tracks with cyclone number > 50. * Per #1714, adding genesis_match_point_to_track config file option for TC-Gen. Note that this version of the code is close but doesn't actually compile yet. I still need to figure out exactly how to process the operational tracks. Should this logic also apply to the matching for those tracks? * Per #1714, the logic for checking the operational tracks is pretty simple. We only store/check operational track points for lead time = 0. So applying the genesis_match_point_to_track boolean config option does not make sense. * Per #1714, update the tc-gen user's guide chapter to describe the updated logic and new config file option. * Per #1714, fix the logic of the is_match() function. * Per #1714, reconfigure the call to tc_gen to exercise the new genesis_match_track_to_point option. * Per #1714, just fixing spacing in source code. * Committing 2 small changes not specifically related to #1714, but related the processing of genesis tracks. When getting items from ATCFGenLines, the columns to be shifted are off by one. We had been shifting offset 2 up to 3, but it should have remained at 2. Also when initializing a TrackInfo object, set the StormID by calling ATCFLineBase::storm_id() instead of constructing it from BASIN:CYCLONE:YYYY. For ATCFGenLines we want to set the Storm ID equal to the 3rd column rather than constructing it! * Per #1714, fix an error in the logic of GenesisInfo::is_match(const GenesisInfo &,...). I was using the index of the current GenesisInfo object instead of the one from the input argument. Fix this by adding GenesisInfo::genesis() member function to return a reference the TrackPoint for Genesis. * Per #1714, correcting logic for parsing the storm_id and warning_time columns for ATCFGen and regular ATCF line types. For ATCFGen line types, the code was incorrectly using the 3rd column when it should have used the 4th column! Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
* Cleaned bash comparison operators; Made changes for MET to compile using GNU 10.1.0 compilers * Updated documentation for new flag for BUFRLIB compilation
…string Bugfix 1755 nccf timelevel by string
* Per #1735, enhance Stat-Analysis to support multiple -out_thresh settings when processing aggregate_stat jobs for MPR line types. This applies to output for FHO, CTC, CTS, ECLV, CNT, SL1L2, and SAL1L2. * Per #1735, since we are writing multiple output line types to the same .stat file, adding stat_row counter to the job class. This way all functions that write to it can increment that row counter when needed. * Per #1735, lots of little changes here to enable the aggregate_stat job type to write multiple output line types.g * Per #1735, update documentation for stat_analysis. * Per #1735, add AsciiTable::expand() function to increase the AsciiTable dimensions and also update the Stat-Analysis handling of the output .stat file it writes. * Per #1735, fix bug in the write_job_aggr_ssvar where I was using the wrong row counter when writing .stat outptu. * Per #1735, print a warning message is the continuous filtering logic results in 0 matched pairs. Also, match existing logic to NOT WRITE any output, not even header rows, when the output AsciiTable contains no results. * Per #1735, update unit_stat_analysis.xml to consolidate jobs stat_analysis_AGG_STAT_ORANK_RHIST and stat_analysis_AGG_STAT_ORANK_PHIST down into 1 job with 2 output line types. Rename the output files accordingly. * Per #1735, update the STATAnalysis config file for processing MPR data by tweaking the jobs to write multiple output line types and/or apply multiple output thresholds.
…When creating the variable to write the temp NetCDF file, we just need to pass through the fill value for the data. Also, make the script less verbose.
…MET. Update all 3 to print consistent log message when writing/reading the temp file. In particular, print the system command that is being executed at Debug(4) to make it easier to replicate python embedding problems that may arise.
…he input in a regular numpy array instead of a masked array.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
Update develop-ref after PR #1754 for issue #1735. This PR modifies unit_stat_analysis.xml to run aggregate_stat jobs which write multiple output thresholds and output line types.
The NB for develop failed on 20210417 and the changes are limited to stat_analysis output as expected:
I inspected the modified files and confirmed that all diffs are expected.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
No additional testing needed.
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
This PR serves as documentation of the change.
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
As described abvoe.
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s), Project(s), and Milestone