apply upper threshold to reflectivity data when calculating DBZH averages

[ghost]

The German Weather Service (DWD) provides DBZH and VRADH values for all German weatherradars in almost real time at its open data server. The data are provided as single sweeps and need to be combined into pvol-files (I do this with wradlib under Python). When calculating vp-files (with function calculate_vp in bioRad) implausibly high DBZH-values are calculated while dbz and others look reasonable:

Regrettably, I couldn't figure out the reason yet. To my surprise, the data are stored as 8-bit unsigned integers in these DWD-datasets:

variables:
ubyte data(360, 720);
:CLASS = "IMAGE";
:IMAGE_VERSION = "1.2";
:_ChunkSizes = 45U, 180U; // uint

group: what {
// group attributes:
:quantity = "DBZH";
:gain = 0.5039525691699605; // double
:offset = -32.50395256916996; // double
:nodata = 255.0; // double
:undetect = 1.7976931348623157E308; // double
}

Could this explain the problems with the DBZH-data in vertical profiles? But why are the dbz presumably correct? Any ideas are welcome!

[kmuehlbauer]

@ohueppop Hi, wradlib dev here. Just want to rule out any problems on the wradlib side. Could you please attach a code snippet how you've created the PVOL-file using wradlib? Normally the DWD DBZH values are unit16, not uint8.

[adokter]

Hi @ohueppop, dbz is biological reflectivity (ground clutter and precipitation removed), while DBZH includes both biology and precipitation. So therefore the two won't be identical, and DBZH is available primarily to visualize precipitation. Could it be that the case shown above included some rain?

[ghost]

Hi @kmuehlbauer, thank you very much for picking this up (and for developing and maintaining the wonderful Python wradlib). The problem is not on the wradlib site. I downloaded the weather radar data from opendata.dwd.de/weather/radar/sites/sweep_vol_v/.../hdf5/filter_simple and opendata.dwd.de//weather/radar/sites/sweep_vol_z/.../hdf5/filter_simple. It was necessary to use the "filter_simple" data because with the change in post-processing by the DWD (in early 2020?) bird densities and migration traffic rates calculated with vol2bird were much lower (and certainly not correct) when using "filter_polarimetric". However, obviously some strong ground clutter (DBZH > 60, hills or windfarms?) "survived" the filtering process for the filter_simple-data. Setting the cells with DBZH values > 40 in the ODIM-file (generated with vol.to_odim in wradlib) to "nodata" (with hdf5.py) resulted in reasonable DBZH values in vol2bird (luckily bird densities were hardly affected). In some of the very recent simple_filter data there are still very high reflectivities. But I solved the problem as mentioned above.

In April 2021 the data format was definitely uint8, now it is uint16 (again?).

[ghost]

Hi @adokter, thanks for your comment. Actually, I use DBZH only to visualize precipation. The issue came up because the high DBZH-values filled large parts of the time-series of vertical profiles plotted with bioRad black. In the example I showed there was no rain at all in the ppis.

[adokter]

Hi @ohueppop, could you upload one polar volume here that has that issue, then I can have a look at what's going on.

[ghost]

Hi @adokter, with pleasure! This is exactly the file (made with wradlib) I used before to clarify the problem:
defld_pvol_stqual_20210401T005500.zip

@kmuehlbauer: When producing these OdimH5-files with vol.to_odim in wradlib the Nyquist intervals (needed for vol2bird) wavelength attributes got lost. I added these to dataset attributes in the just saved OdimH5-files with hdf5.py. Presumably this is rather an issue for the wradlib-users@googlegroups.com. But I need to check first if this problem still exists with recent DWD-files.

[kmuehlbauer]

@ohueppop Great, let me check. Yes, only absolutely necessary attributes are transferred.

Please go to openradar.discourse.group for discussion or https://github.com/openradar/xradar for feature request. We've joined forces in the openradar community in the latter to provide a single radar reading package based an xarray.

[kmuehlbauer]

@ohueppop I can't figure out why this data has dtype uint8. It was after the overall change to 250m resolution, which confuses me a bit. Would you mind sending me at least one of the source files?

@adokter Sorry for iterating that problem here. If you think it's not appropriate I'll create an issue on the wradlib tracker then.

[ghost]

@kmuehlbauer Here are the source files which I downloaded from opendata.dwd.de and used to produce the pvol-file with wradlib I uploaded three days ago.
defld_202104T010055.zip
Thank you for checking! I could upload from other DWD radar stations from that period, too.

[kmuehlbauer]

@ohueppop OK, it's already in the source files (8bit resolution only):

GROUP "dataset1" {
      GROUP "data1" {
         DATASET "data" {
            DATATYPE  H5T_STD_U8LE
            DATASPACE  SIMPLE { ( 360, 720 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
         }

Nothing we can do about, unfortunately. Do you have the polarimetric filtered data? Is it the same?

[ghost]

@kmuehlbauer No, regrettably I don't.

[adokter]

Hi @ohueppop I checked your file and I obtain these incorrect averaged DBZH values as well. I'll have to dive deeper in the code to fix this, but marking this as a bug.

[ghost]

Hi @adokter I'm pretty sure that, compared to the "polarimetric" files, less strict filtering in the "simple" files is the reason. More birds, but also more clutter 😞

[adokter]

Hi @ohueppop, I see in your polar volume file that it has some speckle in the data with VERY high reflectivities above 80 dBZ, up to 95.5 dBZ. It's these few pixels that mess up the rain reflectivities, because the average DBZH we calculate in the profile is based on a conversion of the pixels to linear values, averaging those, and than back-transforming the average to the logarithmic scale.

For the bird profile (dbz, eta, dens quantities) these high pixels are thrown out automatically, because they are unrealistically high for bird migration. For precipitation we don't have an equivalent upper threshold. Your DBZH data would be cleaned up if reflectivity pixels > 80 dBZ would be discarded, which would be a threshold that would rarely exceeded for precipitation. I'll make this issue such a feature request.