Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance SeriesAnalysis wrapper to allow different field info values for each file in a list #1166

Closed
21 tasks
georgemccabe opened this issue Sep 22, 2021 · 5 comments · Fixed by #1353
Closed
21 tasks

Enhance SeriesAnalysis wrapper to allow different field info values for each file in a list #1166

georgemccabe opened this issue Sep 22, 2021 · 5 comments · Fixed by #1353
Assignees
@georgemccabe @j-opatz
Labels
alert: NEED MORE DEFINITION Not yet actionable, additional definition required component: python wrapper MET: Gridded Analysis Tools METplus: Configuration priority: high High Priority requestor: NOAA/CPC NOAA Climate Prediction Center type: enhancement Improve something that it is currently doing
Milestone
METplus-4.1.0

Comments

@georgemccabe
Copy link
Collaborator

georgemccabe commented Sep 22, 2021 •
edited by j-opatz
Loading

Currently the wrapper will call series_analysis once per field defined. This works fine if all of the files in the file list use the same field information. However, in the case with CPC's NMME file, if a single input file contains multiple fields that should be processed, the wrapper cannot be configured to specify a different level value (for each time).

Describe the Enhancement

The preferred method of handling this would allow a user to list one file for a given field, and use a counter (or something that changes with each run) for the changing field index to access each field in the file. Specifically for NMME data, this would be pulling out the multiple time indices that are contained in the observation files.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2700041

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Linked issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@georgemccabe georgemccabe added type: enhancement Improve something that it is currently doing component: python wrapper priority: high High Priority alert: NEED MORE DEFINITION Not yet actionable, additional definition required alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle METplus: Configuration MET: Gridded Analysis Tools labels Sep 22, 2021
@georgemccabe georgemccabe added this to the METplus-4.1.0 milestone Sep 22, 2021
@georgemccabe georgemccabe self-assigned this Sep 22, 2021
@JohnHalleyGotway
Copy link
Collaborator

@georgemccabe, you and I discussed what changes might be needed for the SeriesAnalysis Wrapper via Slack. And this issue is the result of that discussion. Prior to actually making changes to the wrapper, I'd recommend manually configuring/running Series-Analysis to test/confirm that the proposed wrapper changes will actually have the desired effect. If you have questions or want to work through those tests together, I'm happy to help.

@georgemccabe
Copy link
Collaborator Author

This issue came up in GitHub Discussion #1143

@georgemccabe
Copy link
Collaborator Author

From a Slack message from @JohnHalleyGotway :

If you run series_analysis with n input files and 1 field in the config file, it reads that same field from all n input files. If you configure it with n files and n fields, it should read the n-th field from the n-th file.
So you configure it to define separate fields for each of the n input files.

@georgemccabe
Copy link
Collaborator Author

georgemccabe commented Dec 16, 2021 •
edited
Loading

Enhancements to the wrapper logic that are needed to resolve this issue:

  • Keep time information for files to process available in SeriesAnalysis wrapper so that appropriate time information can be substituted into the field info
  • If field info has 1 item and uses a template tag, sub values using the time info for each file in the list

@j-opatz j-opatz self-assigned this Dec 16, 2021
@j-opatz
Copy link
Contributor

j-opatz commented Dec 16, 2021 •
edited
Loading

During a successful direct call to the series-analysis tool, I used the following command at runtime:

/d1/projects/MET/MET_regression/develop/NBdate +\%Y\%m\%d/MET-develop/met/bin/series_analysis -fcst /d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198201_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198301_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198401_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198501_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198601_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198701_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198801_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_198901_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199001_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199101_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199201_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199301_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199401_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199501_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199601_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199701_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199801_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_199901_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200001_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200101_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200201_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200301_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200401_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200501_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200601_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200701_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200801_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_200901_ens.nc
/d1/projects/CPC_data/output/NMME_out/GenEnsProd-SA/gen_ens_prod_201001_ens.nc
-obs /d1/projects/CPC_data/input/NMME/new_data/ghcn_cams.1x1.1982-2020.mon.nc -out hereReadThisSA.nc -config SAConfig_testing_for_CFSv2 -v 4 -log testing_SArun_WED.log

Also attaching the MET config file that was used with this command.
SAConfig_testing_for_CFSv2.txt

georgemccabe added a commit that referenced this issue Dec 17, 2021
@georgemccabe
per #1166, added logic to check if time template tags are used in the…
d243d33 
… field level and if they are loop through all files in the opposite (fcst/obs) file list, extract the time information, and substitute each time into the level value. This allows the correct field to be pulled from a file that contains multiple times while the other data set has separate files for each time
@TaraJensen TaraJensen removed alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle labels Jan 4, 2022
@j-opatz j-opatz added the requestor: NOAA/CPC NOAA Climate Prediction Center label Jan 4, 2022
@georgemccabe georgemccabe linked a pull request Jan 12, 2022 that will close this issue
Feature 1166 series analysis field info #1353
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@georgemccabe georgemccabe

@j-opatz j-opatz

Labels
alert: NEED MORE DEFINITION Not yet actionable, additional definition required component: python wrapper MET: Gridded Analysis Tools METplus: Configuration priority: high High Priority requestor: NOAA/CPC NOAA Climate Prediction Center type: enhancement Improve something that it is currently doing
Projects
None yet
Milestone
METplus-4.1.0
Development

Successfully merging a pull request may close this issue.

Feature 1166 series analysis field info dtcenter/METplus
4 participants
@JohnHalleyGotway @TaraJensen @georgemccabe @j-opatz