Feature 748 ensemble stat config #775
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tently for all wrappers that use those variables
…s that differ from the default to the appropriate METplus conf file
…se case tests to be more descriptive
… tool (TCRMW) does not set it, removed reading of _REGRID_TO_GRID in wrappers since it is handled in one place now
…nary instead of setting it directly as an environment variable so it is more clear what env var is being set, added test
…NS_THRESH to contain ens_thresh = <value>; instead of just the value
…r set_met_config_string/bool
…ult value may not be NA, added more tests for set_met_config_* functions
…nst MET variables
…o another directory
… instead of MET config
|
On kiowa, I ran 3.1 code on use cases (develop on WOFS case that is not in 3.1 codebase) then ran feature branch on use cases. I then ran the NetCDF diffing utility in ci/utils/netcdf_util.py to compare the NetCDF output files from each run. Before data: After data: A few differences in use case results were reconciled with METplus config updates. The WOFS output was different due to a hard-coded random seed value in the MET config file for that use case. I was able to replicate the output by changing this value. I have not yet compared the stat output. |
|
I modified the comparison scripts to compare files that are not NetCDF with the python filecmp utility. I verified that all of the use cases except the cam-multi output have identical output (For now I skip the log directory and metplus_final.conf file). The file lists differ because the paths of the files in them are in different directories, which I can add a check for (if file starts with file_list, then replace dir B with dir A for comparison). The EnsembleStat output files for cam-multi are all empty in the feature branch version, so I will investigate that. |
…sts that have different directories in them by replacing dir B with dir A in one of them
|
cam-multi output was missing because the wrong message_type value was set in the METplus config file. The runs now produce identical output. I also enhanced the directory diffing logic to handle file_lists properly. This PR is ready for review. |
21 tasks
|
For reference, here are the commands I used to run the use cases: and the logic to perform the diffs |
JohnHalleyGotway
approved these changes
Jan 25, 2021
21 tasks
21 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
Updated use case config files to use the new environment variables.
Verified use cases ran successfully.
Review code changes.
Download the output data from GitHub Actions before and after the changes and verify that the output from the EnsembleStat use cases are the same.
Do these changes include sufficient documentation and testing updates? [Yes]
Additional documentation will be handled in Update setting of environment variables for MET config files to add support for all to METPLUS_ vars #768
Will this PR result in changes to the test suite? [Yes]
If yes, describe the new output and/or changes to the existing output:
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s), Project(s), and Milestone