Nonsense values with chgres_cube using GRIB2 files with duplicate records

[weather4evr]

When trying to run chgres_cube with GFS GRIB2 files, there were some nonsense values in the output that crashed FV3 immediately upon attempted integration.

I think I found the problem:

GFS GRIB2 data are across two files:
gfs.t00z.pgrb2.0p25.f000
gfs.t00z.pgrb2b.0p25.f000

I combine them to get higher vertical resolution:
cat gfs.t00z.pgrb2.0p25.f000 gfs.t00z.pgrb2b.0p25.f000 > atmanl.grib2

I send the combined file (atmanl.grib2) to chgres_cube.

Turns out that there is some duplication between gfs.t00z.pgrb2.0p25.f000 and gfs.t00z.pgrb2b.0p25.f000. Specifically, variables at 1, 2, 3, 5, and 7 hPa (for HGT, TMP, RH, UGRD, VGRD, ABSV, O3MR) are in both GFS files (there are also 6 more duplicate records I've not tracked down). When "cat"ting the two files together, there are indeed duplicate records in the catted file (atmanl.grib2, per above example).

chgres_cube does not like the GRIB2 file with duplicate records at 1, 2, 3, 5, and 7 hPa. With the duplicates, the program gives nonsense toward the top of the model (between levels 1-23). This definitely seems like a bug that should be fixed.

Fortunately, removing the duplicate records from the catted GRIB2 file leads to success.

Craig Schwartz

[JeffBeck-NOAA]

@GeorgeGayno-NOAA, @LarissaReames-NOAA, have you run into this issue before?

[LarissaReames-NOAA]

Yeah I think we encountered similar nonsense with some catted files that EMC had been using which also had duplicate entires near the top of the model. I think the desired behavior here should be to quit with an error if duplicate entries are encountered -- I would think finding a way to just pick one of the duplicates wades in to dangerous territory, running the risk of assuming the data in the duplicate entries are the same when they might not be (like in the case of a mislabeled variable). Thoughts?

[GeorgeGayno-NOAA]

What is the date of this grib2 data?

[weather4evr]

Same issue for both 2023012500 and 2023012600

[GeorgeGayno-NOAA]

@weather4evr What fields and levels had nonsensical values?

[weather4evr]

"zh" was nonsense from the top of the model (level 1) to level 23. all other 3d fields were nonsense from levels 12-23.

[LarissaReames-NOAA]

@weather4evr Do you have a copy of the Frankenstein file you created that had the duplicate levels somewhere I could grab it from?

[weather4evr]

Yup. I just pushed it to jet:

/home/Craig.Schwartz/atmanl.grib2

It's from today (2023012700), which causes the same problem.

[GeorgeGayno-NOAA]

Perhaps some wgrib2 tricks could be used to combine the files such that there are no duplicate records. That might be easier than modifying the code. A script that combines the files could be added to the util directory.

[JiliDong-NOAA]

Just FYI. I believe EMC has been using the combined GFS grib2 for LBCs in parallel and experimental RRFS for a while. One more step is always taken to remove duplicate records after the concatenation.

The duplicate records removal follows:
https://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/tricks.wgrib2
entry (29): How do I remove duplicate records?

wgrib2 IN.grb -submsg 1 | unique.pl | wgrib2 -i IN.grb -GRIB OUT.grb

[weather4evr]

@JiliDong-NOAA That's exactly what I did to get things to work.

At the very least, if there's not an elegant way to deal with this issue in the chgres_cube code, can the documentation be updated on the web to alert users of this issue?

[GeorgeGayno-NOAA]

Just FYI. I believe EMC has been using the combined GFS grib2 for LBCs in parallel and experimental RRFS for a while. One more step is always taken to remove duplicate records after the concatenation.

The duplicate records removal follows: https://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/tricks.wgrib2 entry (29): How do I remove duplicate records?

wgrib2 IN.grb -submsg 1 | unique.pl | wgrib2 -i IN.grb -GRIB OUT.grb

@JiliDong-NOAA, would @BenjaminBlake-NOAA be familiar with the details?