Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bufr2ioda_snocvr_bufr.py fails because expected input not present #907

Closed
RussTreadon-NOAA opened this issue Feb 5, 2024 · 19 comments · Fixed by #937
Closed

bufr2ioda_snocvr_bufr.py fails because expected input not present #907

RussTreadon-NOAA opened this issue Feb 5, 2024 · 19 comments · Fixed by #937

Comments

@RussTreadon-NOAA
Copy link
Contributor

Global-workflow job gdasprepatmiodaobs failed for 2021122018 on Hera

wxflow.executable.ProcessError: Command exited with status 9:
'/scratch1/NCEPDEV/da/role.jedipara/git/global-workflow/atmos/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_snocvr_bufr.py' '-c' '/scratch1/NCEPDEV/stmp2/role.jedipara/RUNDIRS/prc\
i/prepatmobs.209061/snocvr_bufr_2021122018.json'
+ JGLOBAL_ATM_PREP_IODA_OBS[1]: postamble JGLOBAL_ATM_PREP_IODA_OBS 1707157080 1

The converter failed because expected input file /scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr.tm00.bufr_d does not exist.

forrtl: Permission denied
forrtl: severe (9): permission to access file denied, unit 12, file /scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr.tm00.bufr_d
Image              PC                Routine            Line        Source
libifcoremt.so.5   00002B288B696E79  for__io_return        Unknown  Unknown
libifcoremt.so.5   00002B288B6BCE45  for_open              Unknown  Unknown
libbufr_4.so       00002B2885BACF78  open_f                     98  bufr_c2f_interface.F90
bufr.cpython-37m-  00002B28851B4248  _ZN8Ingester4bufr          24  NcepDataProvider.cpp
bufr.cpython-37m-  00002B28851BB323  _ZN8Ingester4bufr          35  File.cpp
bufr.cpython-37m-  00002B2885241F52  _ZZN8pybind1112cp          76  init.h
bufr.cpython-37m-  00002B288519E80B  _ZN8pybind1112cpp         946  pybind11.h
python3            000055BC751C6934  _PyMethodDef_RawF     Unknown  Unknown
python3.7          000055BC7522B9CA  Unknown               Unknown  Unknown
python3            000055BC751AE9DC  PyObject_Call         Unknown  Unknown
python3.7          000055BC7522BFEE  Unknown               Unknown  Unknown
python3.7          000055BC751AE2EA  Unknown               Unknown  Unknown
bufr.cpython-37m-  00002B2885198B8A  pybind11_meta_cal         187  class.h

...

Hera(hfe09):/scratch1/NCEPDEV/da/role.jedipara/para_gfs/prci$ ls -l /scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr.tm00.bufr_d
ls: cannot access /scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr.tm00.bufr_d: No such file or directory

Dump files can be missing in operations. Script bufr2ioda_snocvr_bufr.py along with other bufr2ioda converters need to account for this possibility.
@RussTreadon-NOAA
Copy link
Contributor Author

Tagging @jiaruidong2017 for awareness

@jiaruidong2017
Copy link
Collaborator

Tagging @YoulongXia-NOAA

@RussTreadon-NOAA
Copy link
Contributor Author

Tagging @YoulongXia-NOAA

Thanks @jiaruidong2017

@jiaruidong2017
Copy link
Collaborator

@RussTreadon-NOAA As I know, the operational gdas.t{HH}z.snocvr.tm00.bufr_d data is available from 2022-12-03.

@RussTreadon-NOAA
Copy link
Contributor Author

Do you or @YoulongXia-NOAA have access or can you create a snocvr file for 2021122018. I'm working on this date since it is the date for g-w CI case C96_atmsnowDA

@YoulongXia-NOAA
Copy link
Collaborator

@jiaruidong2017 and @RussTreadon-NOAA, I used 20230501 00z file as an input example as after May 2023, the snocvr bufr data includes MADIS snow depth data. This is a new ops product and Jiarui will use it in near future. To avoid this case, this may need a flag to flag out for use of before May 2023.

@RussTreadon-NOAA
Copy link
Contributor Author

@YoulongXia-NOAA , good suggestion. Can you look into implementing this suggestion in GDASApp, g-w, or both?

@jiaruidong2017
Copy link
Collaborator

@RussTreadon-NOAA You can use the file below for your test:

/scratch1/NCEPDEV/global/glopara/dump/gdas.20211221/00/atmos/gdas.t00z.snocvr_snow.nc4

@jiaruidong2017
Copy link
Collaborator

Should be this one as below:

/scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr_snow.nc4

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you @jiaruidong2017. Is file /scratch1/NCEPDEV/global/glopara/dump/gdas.20211220/18/atmos/gdas.t18z.snocvr_snow.nc4 the output of bufr2ioda_snocvr_bufr.py?

If "yes", this is OK for personal testing but it not OK for g-w CI testing.

g-w CI tests the workflow. prepatmiodaobs is part of the workflow. We can't skip this job in g-w CI testing. The way GDASApp currently works we need snocvr.tm00.bufr_d, otherwise bufr2ioda_snocvr_bufr.py fails.

@jiaruidong2017
Copy link
Collaborator

@RussTreadon-NOAA It is the output pre-generated by @YoulongXia-NOAA

The current G-W doesn't support to use the operational snocvr.tm00.bufr_d data yet. Instead, G-W supports to use the preprocessed historical snocvr_snow.nc4 output. For your CI test, it should be okay with this output file. You just need to skip the bufr2ioda_snocvr_bufr.py step.

@RussTreadon-NOAA
Copy link
Contributor Author

Understood, @jiaruidong2017 . There is currently no mechanism in GDASApp to skip bufr2ioda_snocvr_bufr.py. Hence my opening this issue. We're presently stuck.

@jiaruidong2017
Copy link
Collaborator

jiaruidong2017 commented Feb 5, 2024 via email •
edited
Loading

@YoulongXia-NOAA
Copy link
Collaborator

@jiaruidong2017, I am not familiar with GDASApp, could you please add a flag to skip this test for in GDASApp, g-w, or both? If we cannot do that, we need to pull this converter out from GDASApp right now till when a test after May 01 2023 will be run, and then add it back.

@jiaruidong2017
Copy link
Collaborator

jiaruidong2017 commented Feb 5, 2024 via email •
edited
Loading

@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Feb 8, 2024
Closed
@RussTreadon-NOAA
Copy link
Contributor Author

Add the following to a working copy of /ush/ioda/bufr2ioda/bufr2ioda_snocvr_bufr.py

@@ -46,6 +46,9 @@ def bufr_to_ioda(config, logger):

     bufrfile = f"{cycle_type}.t{hh}z.{data_type}.tm00.{data_format}"
     DATA_PATH = os.path.join(dump_dir, f"{cycle_type}.{yyyymmdd}", str(hh), 'atmos', bufrfile)
+    if not os.path.isfile(DATA_PATH):
+        logger.info(f"DATA_PATH {DATA_PATH} does not exist")
+        return

     logger.debug(f"The DATA_PATH is: {DATA_PATH}")

Ran gdasprepatmiodaobs for 2021032318 with the above change. The following runtime printout was generated for snocvr

2024-02-26 20:00:26,700 - INFO     - run_bufr2ioda.py: Convert snocvr_bufr...
2024-02-26 20:00:26,700 - INFO     - gen_bufr2ioda_json.py: Using /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/parm/gdas/bufr2ioda/bufr2ioda_snocvr_bufr.json as input
2024-02-26 20:00:26,706 - INFO     - gen_bufr2ioda_json.py: Wrote to /scratch1/NCEPDEV/stmp2/Russ.Treadon/RUNDIRS/prci/prepatmobs.160286/snocvr_bufr_2021032318.json
2024-02-26 20:00:26,985 - INFO     - run_bufr2ioda.py: Executing /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_snocvr_bufr.py -c /scratch1/NCEPDEV/stmp2/Russ.Treadon/RUNDIRS/prci/prepatmobs.160286/snocvr_bufr_2021032318.json
2024-02-26 20:00:31,796 - INFO     - bufr2ioda_adpsfc_prepbufr.py: DATA_PATH /scratch1/NCEPDEV/global/glopara/dump/gdas.20210323/18/atmos/gdas.t18z.snocvr.tm00.bufr_d does not exist

Job prepatmiodaobs successfully ran to completion.

Is this an acceptable way to address the situation in in which an expected input bufr dump file is missing?

Tagging @emilyhcliu , @ADCollard , @guillaumevernieres @ShastriPaturi , @CoryMartin-NOAA , and @jiaruidong2017

@CoryMartin-NOAA
Copy link
Contributor

I think this is acceptable for if it is missing. What about if it is corrupted or not a BUFR file? Do we want that case to fail, or to still be handled gracefully?

@RussTreadon-NOAA
Copy link
Contributor Author

Agreed. We need to harden our processing for a variety of operational possibilities:

  • corrupted BUFR file
  • not a BUFR file
  • zero length BUFR file
  • valid BUFR file not containing any of the queried subsets. This happened with bufr2ioda_satwind_amv_goes.py. The subsets specified in bufr2ioda_satwind_amv_goes.json were not in a retrospective satwnd bufr file.
  • etc

@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Feb 27, 2024
Closed
@RussTreadon-NOAA
Copy link
Contributor Author

We should turn on JEDI ATM only CI testing in g-w (see issue #2294). Doing so requires that we resolve bufr2ioda issues ( #897, #898, #907).

RussTreadon-NOAA added a commit to RussTreadon-NOAA/GDASApp that referenced this issue Feb 27, 2024
@RussTreadon-NOAA
check for existence of snocvr bufr dump file (NOAA-EMC#907)
f63f66a
@RussTreadon-NOAA RussTreadon-NOAA mentioned this issue Feb 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enhance error checking for bufr2ioda converters RussTreadon-NOAA/GDASApp
4 participants
@CoryMartin-NOAA @RussTreadon-NOAA @jiaruidong2017 @YoulongXia-NOAA