Provide bug fixes and add ALTKOB json file: marine monthly in situ BUFR #879

ShastriPaturi · 2024-01-25T19:38:44Z

This PR does two things:

Fixes bugs in the marine monthly in situ BUFR python converters
Ads json file for ALTKOB surface obs

fixes #877

unwanted file

ShastriPaturi · 2024-01-26T00:15:35Z

@emilyhcliu, the 6-hrly dumps for 20210801 are on orion at:
/work2/noaa/da/spaturi/forRuss/gdas.20210801

Let me know, if you can access it. It does have altkob bufr data as well.

emilyhcliu · 2024-01-26T17:55:30Z

For booking purpose, here is the list of marine-related data types added to bufr2ioda processing:

altkob_surface
bathythermal_profiles
subpfl_argo_profiles
subpfl_glider_profiles
tesac_mamals
tesac_profiles
tackob_surface
xbtctd_profiles

emilyhcliu · 2024-01-26T18:14:24Z

Marin data processing were tested with the latest feature branch ( bugfix/monthly_marine_bufr2ioda) for prepatmiodaobs step

The ROTDIR for the test on Orion: /work2/noaa/da/eliu/gdas-validation/comroot/gdas_eval_sevcsr_JEDI/gdas.20210801/00/obs

Summary of test results:
1. The following two types ran bufr2ioda successfully; IODA files created successfully

bathythermal_profiles
tesac_profiles

2. Trackob_surface failed in python (see below)

tackob_surface

Traceback (most recent call last):
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_trackob_surface.py", line 277, in <module>
    bufr_to_ioda(config, logger)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_trackob_surface.py", line 106, in bufr_to_ioda
    dateTime = r.get_datetime('year', 'month', 'day', 'hour', 'minute', group_by='depth')
IndexError: vector::_M_range_check: __n (which is 15) >= this->size() (which is 15)

3. The following three types received the following error message:

tesac_mamals
subpfl_argo_profiles
subpfl_glider_profiles
altkob_surface
xbtctd_profiles

forrtl: severe (9): permission to access file denied, unit 12, file /work/noaa/rstprod/dump/gdas.20210801/00/atmos/gdas.t00z.subpfl.tm00.bufr_d
forrtl: severe (9): permission to access file denied, unit 12, file /work/noaa/rstprod/dump/gdas.20210801/00/atmos/gdas.t00z.altkob.tm00.bufr_d

I updated the obs directory in ROTDIR with the bufr_d files from /work2/noaa/da/spaturi/forRuss/gdas.20210801/00/atmos, where all required marine data (total 8) exist. However, the bufr2ioda processing still looking for the global dump (/work/noaa/rstprod/dump/gdas.20210801/00) for input.

The obs ROTDIR for the gdasprepatmiodaob is here:
/work2/noaa/da/eliu/gdas-validation/comroot/gdas_eval_sevcsr_JEDI/gdas.20210801/00/obs

Investigating....
OK. I thnk I know what's going one.
The json files need to be modified. I am testing now. Will report here shortly.

Added depth

ShastriPaturi · 2024-01-26T19:35:52Z

@emilyhcliu, the bufr2ioda_trackob_surface.py was missing the depth variable.
It should be fine now.

emilyhcliu · 2024-01-26T22:38:20Z

To do the test correctly, I have created a temporary dump for 2021080100
The temporary dump is located at /work/noaa/stmp/eliu/dump
This dump contains 2021080100 data from the official dump and Shastri's marine data dump

The DMPDIR in the config.base is assigned to the temporary dump: /work/noaa/stmp/eliu/dump

There are total 8 bufr2ioda processings for marine data:
altkob_surface
bathythermal_profiles
subpfl_argo_profiles
subpfl_glider_profiles
tesac_mamals
tesac_profiles
tackob_surface
xbtctd_profiles

6 Out of the 8 marine data processings, 6 of them completed successfully:

-rw-r----- 1 eliu da    872371 Jan 26 16:03 gdas.t00z.argo_profiles.subpfl.nc
-rw-r----- 1 eliu da     98862 Jan 26 16:03 gdas.t00z.bathy_profiles.bathy.nc
-rw-r----- 1 eliu da    459087 Jan 26 16:03 gdas.t00z.glider_profiles.subpfl.nc
-rw-r----- 1 eliu da    460361 Jan 26 16:03 gdas.t00z.tesac_profiles.tesac.nc
-rw-r----- 1 eliu da     20127 Jan 26 16:03 gdas.t00z.trackob_profiles.trkob.nc
-rw-r----- 1 eliu da     24060 Jan 26 16:03 gdas.t00z.xbtctd_profiles.xbtctd.nc

The following 2 data types failed:

altkobs_surface.
tesac_mammals_profiles

Here are the error messages related to the failures:
altkobs_surface

Traceback (most recent call last):
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_altkob_surface.py", line 275, in <module>
    bufr_to_ioda(config, logger)

  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_altkob_surface.py", line 106, in bufr_to_ioda
    dateTime = r.get_datetime('year', 'month', 'day', 'hour', 'minute', group_by='depth')
IndexError: vector::_M_range_check: __n (which is 14) >= this->size() (which is 14)

tesac_mammals_profiles

Traceback (most recent call last):
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_tesac_mammals_profiles.py", line 315, in <module>
    bufr_to_ioda(config, logger)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/sorc/gdas.cd/ush/ioda/bufr2ioda/bufr2ioda_tesac_mammals_profiles.py", line 172, in bufr_to_ioda
    logger.debug(f" temp          min, max, length, dtype = {temp.min()}, {temp.max()}, {len(temp)}, {temp.dtype}")
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/site-packages/numpy/ma/core.py", line 5701, in min
    axis=axis, out=out, **kwargs).view(type(self))
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/site-packages/numpy/core/_methods.py", line 44, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity

Removed depth, stationID

ShastriPaturi · 2024-02-01T21:06:21Z

@emilyhcliu, I removed the depth variable with was the cause of the problem in [bufr2ioda_altkob_surface.py].
However, I was able to successfully generate the ioda output with bufr2ioda_tesac_mammals_profiles.py for 2021080100 with no changes to the bufr2ioda_tesac_mammals_profiles.py and the json file generated by run_bufr2ioda.py.

If it still is a problem, may I suggest you comment out or not to test bufr2ioda_tesac_mammals_profiles.py?
Also, thanks a lot for identifying bugs in the python converters.

RussTreadon-NOAA · 2024-02-27T14:41:22Z

@ShastriPaturi : Will this PR, #879, or #914 fix issue #907. It would be good to turn on JEDI DA ATM CI testing in g-w (issue #2294). Doing so requires that marine bufr2ioda converters to not fail when expected input bufr files are not present.

ShastriPaturi · 2024-02-27T15:39:38Z

@ShastriPaturi : Will this PR, #879, or #914 fix issue #907. It would be good to turn on JEDI DA ATM CI testing in g-w (issue #2294). Doing so requires that marine bufr2ioda converters to not fail when expected input bufr files are not present.

@RussTreadon-NOAA yes.

RussTreadon-NOAA · 2024-02-27T15:43:45Z

Great! This is good news @ShastriPaturi. When do you expect #879 and #914 to be merged into GDASApp develop? The timeline for these PRs impacts the timeline for g-w issue #2294.

ShastriPaturi · 2024-02-28T19:07:08Z

Great! This is good news @ShastriPaturi. When do you expect #879 and #914 to be merged into GDASApp develop? The timeline for these PRs impacts the timeline for g-w issue #2294.

@RussTreadon-NOAA I expect this to be done this week

RussTreadon-NOAA · 2024-02-28T19:11:51Z

@ShastriPaturi , are you OK with merging PR #937 into develop before this PR?

guillaumevernieres · 2024-02-28T19:17:18Z

@ShastriPaturi , are you OK with merging PR #937 into develop before this PR?

@RussTreadon-NOAA , I'll speak for @ShastriPaturi and say yes, merge #937 before this PR.

ShastriPaturi · 2024-02-28T19:23:32Z

@ShastriPaturi , are you OK with merging PR #937 into develop before this PR?

@RussTreadon-NOAA , I'll speak for @ShastriPaturi and say yes, merge #937 before this PR.

Yes @RussTreadon-NOAA

ShastriPaturi added 3 commits January 25, 2024 13:05

copy subsampled BUFR to DMPDIR

d9d9483

bugfix

210cfb3

Add altkob json file

eaceb2f

ShastriPaturi marked this pull request as draft January 25, 2024 19:38

ShastriPaturi mentioned this pull request Jan 25, 2024

[BUG] BUFR2IODA conversion problem for altkob data type due to missing associated json file #877

Open

Delete test/soca/gw/setup_obsprep.sh

533b2fb

unwanted file

emilyhcliu self-requested a review January 25, 2024 21:41

Update bufr2ioda_trackob_surface.py

f6c8775

Added depth

Update bufr2ioda_altkob_surface.py

e6d775b

Removed depth, stationID

ShastriPaturi mentioned this pull request Feb 1, 2024

marine bufr2ioda converters fail in prepatmioda job #897

Closed

ShastriPaturi marked this pull request as ready for review February 1, 2024 21:08

ShastriPaturi requested review from guillaumevernieres and removed request for guillaumevernieres February 1, 2024 21:09

RussTreadon-NOAA mentioned this pull request Feb 27, 2024

Enhance error checking for bufr2ioda converters #937

Merged

guillaumevernieres closed this May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide bug fixes and add ALTKOB json file: marine monthly in situ BUFR #879

Provide bug fixes and add ALTKOB json file: marine monthly in situ BUFR #879

ShastriPaturi commented Jan 25, 2024 •

edited

Loading

ShastriPaturi commented Jan 26, 2024 •

edited

Loading

emilyhcliu commented Jan 26, 2024

emilyhcliu commented Jan 26, 2024 •

edited

Loading

ShastriPaturi commented Jan 26, 2024

emilyhcliu commented Jan 26, 2024

ShastriPaturi commented Feb 1, 2024 •

edited

Loading

RussTreadon-NOAA commented Feb 27, 2024

ShastriPaturi commented Feb 27, 2024

RussTreadon-NOAA commented Feb 27, 2024

ShastriPaturi commented Feb 28, 2024

RussTreadon-NOAA commented Feb 28, 2024

guillaumevernieres commented Feb 28, 2024

ShastriPaturi commented Feb 28, 2024

Provide bug fixes and add ALTKOB json file: marine monthly in situ BUFR #879

Provide bug fixes and add ALTKOB json file: marine monthly in situ BUFR #879

Conversation

ShastriPaturi commented Jan 25, 2024 • edited Loading

ShastriPaturi commented Jan 26, 2024 • edited Loading

emilyhcliu commented Jan 26, 2024

emilyhcliu commented Jan 26, 2024 • edited Loading

ShastriPaturi commented Jan 26, 2024

emilyhcliu commented Jan 26, 2024

ShastriPaturi commented Feb 1, 2024 • edited Loading

RussTreadon-NOAA commented Feb 27, 2024

ShastriPaturi commented Feb 27, 2024

RussTreadon-NOAA commented Feb 27, 2024

ShastriPaturi commented Feb 28, 2024

RussTreadon-NOAA commented Feb 28, 2024

guillaumevernieres commented Feb 28, 2024

ShastriPaturi commented Feb 28, 2024

ShastriPaturi commented Jan 25, 2024 •

edited

Loading

ShastriPaturi commented Jan 26, 2024 •

edited

Loading

emilyhcliu commented Jan 26, 2024 •

edited

Loading

ShastriPaturi commented Feb 1, 2024 •

edited

Loading