Diagnosing or specifying missing data value

[mpm-meto]

We have gridded observation files in netcdf which use an underscore as the missing data indicator. It is really unclear what MET does with these. In wavelet_stat for example it seems to convert them to a number. Here the tiles don't include these regions so it's not an issue, but it did make me wonder. I tried to find where you could tell MET what the missing data indicators are but failed to find anything definitive in the user guide. Maybe this can be done from METplus. I have not investigated this yet. I probably haven't been looking in the right place. It may also be that it is diagnosed on reading in, but if the missing data is not set in the netcdf header, I am still not sure how. Based on the file I've attached the value isn't set in the header. Any thoughts on this?

gpm_2019083100_2019090100.nc.gz

[JohnHalleyGotway]

@mpm-meto I suspect that the underscore you're seeing is a red herring. You think it's being stored as an underscore because that's what ncdump is showing you. I believe that generally ncdump displays bad data values as '_' in the output.

For example, when you run "make test", this NetCDF output file is created:
out/pcp_combine/sample_obs_2005080712V_12A.nc

And it has a lot of missing data values around the edge, seen as white in this ncview image:

Running this ncdump command you'll see those values represented as underscores in the output:

ncdump -v APCP_12 out/pcp_combine/sample_obs_2005080712V_12A.nc | more
netcdf sample_obs_2005080712V_12A {
...
        float APCP_12(lat, lon) ;
...
                APCP_12:_FillValue = -9999.f ;
...
data:

 APCP_12 =
  _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, 
...

So what's going on here? Internally, MET stores bad data as a value of "-9999". Whenever it writes NetCDF output files, it writes that value of -9999 to the output, but it also adds the "_FillValue" variable attribute to indicate that that value should be interpreted as bad data. When ncdump processes this file, it interprets the "_FillValue" variable attribute and instead of printing out a bunch of "-9999" values, it prints the much more concise underscore.

To prove this, I removed the _FillValue variable attribute and reran ncdump. See below:

ncatted -a _FillValue,APCP_12,d,f,0 sample_obs_2005080712V_12A.nc sample_obs_2005080712V_12A_no_fill.nc
ncdump -v APCP_12 sample_obs_2005080712V_12A_no_fill.nc | more
netcdf sample_obs_2005080712V_12A_no_fill {
...
	float APCP_12(lat, lon) ;
...
data:

 APCP_12 =
  -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, 

And ncdump now reports "-9999" as a real data value (instead of an underscore) because it no longer knows that that's bad data.

The _FillValue and missing_value attributes are both handled in this way across the gridded NetCDF data that MET reads. It's mentioned in the CF conventions HERE.

Note that you do NOT need to use "-9999" as the missing data value for input into MET. Instead, use whatever you'd like but make sure that that value is listed in the _FillValue or missing_value variable attributes. When reading input data, MET automatically replaces any instances of that missing data value with "-9999" internally.