Consolidate tasks that retrieve observations #850
Assignees
Labels
enhancement
New feature or request
Comments
14 tasks
MichaelLueken
pushed a commit
that referenced
this issue
Aug 15, 2023
As described in Issue #850, the scripts for retrieving observation data for verification are quite convoluted and hard-to-maintain due to the constant changing of naming conventions on HPSS. We can leverage the functionality of the retrieve_data.py script to greatly simplify and consolidate this code, and that's what this PR achieves. In addition, many other bug fixes and improvements are included: * The j-Jobs and exscripts for the retrieving of CCPA, MRMS, NDAS, and NOHRSC data are consolidated into a single set of scripts. * Verification tests that use staged data on disk no longer need to manually deactivate "get_obs" tasks; now tasks check if staged data was specified and ensure it is all present * Obs locations (CCPA_OBS_DIR, MRMS_OBS_DIR, NDAS_OBS_DIR, and NOHRSC_OBS_DIR) no longer need a specific directory structure, though the file format remains the same * Major improvements to mrms_pull_topofhour.py; now has a main() function for importing, uses proper command-line arguments with argparse, uses f-strings to simplify print statements, and raises an exception if no observation file is found instead of failing silently * Fixes bug in retrieve_data.py where filenames in tarballs containing full paths are not properly extracted * Removes "choices" for --data_type argument in retrieve_data.py: the data type is customizable via the config file (which is also customizable), so we shouldn't hard-code arbitrary choices. Missing data types are gracefully handled later anyway. * Add all verification tests to a coverage suite * Add new test custom_ESGgrid_Great_Lakes_snow_8km that specifies a custom domain for a snowstorm in February 2023, re-using data that was staged for the AQM test. This is part of an effort to expand testing that will be completed in a subsequent PR, including explicit tests for winter precipitation verification. * Add cold-air-damming test to all comprehensive test lists, not just orion. * Test MET_ensemble_verification_only_vx_time_lag now works on most platforms, not just hard-coded to Hera * Comments related to verification obs in config_defaults.yaml are consolidated and relevant information moved to new consolidated obs-pulling script
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Currently there are several workflow scripts that retrieve various observations in different ways:
Solution
Utilize the "retrieve_data.py" script for tasks that retrieve data, and consolidate code/scripts where possible.
Requirements**
Utilize the "retrieve_data.py" script for tasks that retrieve data, and consolidate code/scripts where possible.
Acceptance Criteria (Definition of Done)
Utilize the "retrieve_data.py" script for tasks that retrieve data, and consolidate code/scripts where possible.
The text was updated successfully, but these errors were encountered: