Review and update WE2E testing suite

[jwolff-ncar]

• Come up with a full plan for what tests need to be run, where, when, and by whom, in addition to adding new tests when necessary
• Add proper checks for input/output files, output difference testing

[mkavulich]

I originally opened this as its own issue but it is better contained here.

Currently there is an infeasible number of tests contained in regional_workflow/tests/baseline_configs. Even attempting to run most of them not only uses thousands of core hours and takes many wallclock hours even in periods of low machine usage, it often results in errors due to limits on the number of jobs a user can submit at one time:

05/13/21 20:36:30 UTC :: FV3LAM_wflow.xml :: sbatch: error: AssocMaxSubmitJobLimit sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

Nearly the same set of capabilities should be testable with a small subset of all these tests, helping mitigate these issues.

[mkavulich]

#490 is also relevant here, though I'm not sure if that fix would be in the test parameters or the workflow/code

[llpcarson]

Also update RTs for the ufs-weather-model, to avoid duplication and ensure coverage of model configuration tests.

[mkavulich]

#531 Adds descriptions of each test and separates them into categories, this is a good first step to narrowing down the required test set for PRs

[mkavulich]

Superseded by ufs-community/ufs-srweather-app#277