Feature 768 regression tests for use cases

[georgemccabe]

Note: some changes for #768 are in this PR. These changes have been tested and verified that they work properly. I tried to avoid creating extra work to separate out the changes to implement the regression testing logic from that work.

I created a develop-ref branch from the source branch of this pull request. Branches ending with "-ref" trigger the creation of Docker data volumes that are used as "truth" data to diff the output from a pull request.

Pull Request Testing

• Describe testing already performed for these changes:
  

Tested the logic using feature_768_test_regression-ref to test generation of truth data, then created a pull request from feature_768_test_pr into feature_768_test_regression to test that diffing logic executes properly.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

• Review code changes

• Verify that Docker data volumes for output data exist for all use case groups. They should all start with output-develop and are named based on the use case category and any subset numbers (if applicable). There should be 13 of them (matches the number of use cases groups run in the matrix)

https://hub.docker.com/repository/docker/dtcenter/metplus-data-dev/tags

• Review GitHub Actions log output to ensure everything ran as expected

  • Log for creating truth data volumes from develop-ref is here:
    https://github.com/dtcenter/METplus/runs/1923380457?check_suite_focus=true

  • Log for pull request diff testing can be found under Actions tab (look for run that mentioned it is a pull request). The output diff logs are found in the same log as running each use case group. The use case job will turn red if any use case fails or if any differences are found in the output.

Note: Output from GempakToCF creates files that have a fill value of -9999 but contains many data points that are NaN. This caused issues in the diffing logic, so I changed it to output a WARNING if this case occurs. I previously added logic to loop over every data point and compare them if they are both not NaN values, but it increased execution time greatly. This could be revisited later to improve the diffing logic.

• Do these changes include sufficient documentation and testing updates? [Yes]

• Will this PR result in changes to the test suite? [Yes]
  

It makes the test suite much more robust!

Pull Request Checklist

See the METplus Workflow for details.

• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s), Project(s), and Milestone
• After submitting the PR, select Linked Issues with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

I approve of these changes. Details below.

These changes look great and represent a huge improvement to our automated testing. As discussed during the METplus project meeting on 2/18/21, recommend considering the following enhancements:

• When differences are detected, add the files from the reference "truth" branch to the artifact created for that pull request GHA. That'll make it really easy for the PR reviewer to investigate the diffs.
• Can we make the docker data volumes more self-describing? If possible, add a "/README" file that lists info like the creation date, source branch or SHA. There's potential to easily lose track of what data volumes were created by what version of the code and when.
• Recommend adding a mechanism to list files that should be EXCLUDED from the diffing logic. Could be as simple as a text file listing files to be skipped. I'm sure this will come up and be useful.

These changes do NOT need to be made on this branch but should be considered for the future.

• As directed, I checked the dtcenter/metplus-data-dev tags and confirmed that 13 are present, one for each of the groups of tests.
• Pulled one and confirmed the data looks reasonable:

docker run -it --rm dtcenter/metplus-data-dev:output-develop-met_tool_wrapper ls /data/truth

• Reviewed the code changes and there are tons of them... 82 files. Lots and lots of changes included in this PR.
  • Changes for automation.
  • Changes to METplus python wrappers.
  • Changes to METplus conf files.
  • Changes to MET config files.

I reviewed some of the changes, but don't fully appreciate all of them. Recommend in the future trying to break down PR's into smaller, more self-contained pieces.