v4.5.0 does not use NC_FILL_INT for undefined values when expanding variables

[cwardgar]

This is very likely related to the other _FillValue issues that are currently open, such as #458, but I'm not sure. In any event, a test in thredds that passed when using libnetcdf 4.4.1 now fails with 4.5.0.

TestNc4IospWriting.expandUnlimitedDimensions() starts with a 1x1 variable and expands it using nc_put_vars_int, one row or column at a time. Some values will be undefined and it's expected that they will be filled with NC_FILL_INT. Instead, those values are set to 0.

Is that the expected behavior?

[WardF]

I will take a look at this shortly. Is there a way you can test against master? We've addressed a handful of _FillValue issues (reverting changes that were meant to enforce the spec but instead caused breakage in instances where nobody was adhering to the spec, for no good reason). The failure you're seeing is no doubt related to one of these, but off the top of my head I'm not sure if it has been reversed. I'll dig into it, but I'm also at AGU next week.

[cwardgar]

I just built and linked against master, and the problem remains.

[edhartnett]

Do we have any test code for this?

I believe these problems are also at play in the PIO library. I believe that some of these fill value problems remain.

Fill value bugs are very serious because people can be creating data files and getting bad data because fill values are being incorrectly written. And it's almost impossible to detect in the data, in many cases.

I will see about adding some more tests for fill values.

[edhartnett]

I wrote some test code and ran it against master. It checks for fill values in an expanded unlimited dimension.

I found no unexpected behavior.

Were you using more than one unlimited dimension? I have not tried that yet...

[cwardgar]

@edhartnett Yes, the variable had 2 unlimited dimensions. You can find my test here. It's Java, but pretty well-commented.

[edhartnett]

@cwardgar I have tried to reproduce that test in C, but everything is working for me. For example:

For example, this code:

      int varid, ncid, dims[NDIMS2];
      int datum = TEST_VAL_42;
      size_t index[NDIMS2] = {0, 2};
      size_t index2[NDIMS2] = {1, 0};

      if (nc_create(FILE_NAME, NC_NETCDF4 | NC_CLOBBER, &ncid)) ERR;
      if (nc_def_dim(ncid, X_NAME, NC_UNLIMITED, &dims[0])) ERR;
      if (nc_def_dim(ncid, Y_NAME, NC_UNLIMITED, &dims[1])) ERR;
      if (nc_def_var(ncid, VAR_NAME2, NC_INT, NDIMS2, dims, &varid)) ERR;

      /* Write one value. */
      if (nc_put_var1_int(ncid, 0, index, &datum)) ERR;
      if (nc_put_var1_int(ncid, 0, index2, &datum)) ERR;

      if (nc_close(ncid)) ERR;

Produces this output, as expected:

netcdf tst_vars4 {
dimensions:
	x = UNLIMITED ; // (2 currently)
	y = UNLIMITED ; // (3 currently)
variables:
	int var2(x, y) ;
data:

 var2 =
  {_, _, 42},
  {42, _, _} ;
}

Which is what I expect. Note the fill values, indicated by the "_" in ncdump. I have tried other varients of this test, but they all seem to work as expected.

Can you try to get the current master and confirm the problem still exists? Can you suggest what your test is doing that my test is not doing?

[cwardgar]

So this is really strange, but I can no longer reproduce the bug on v4.5.0+. Not on my MacBook or on Linux. I have no idea what happened, because I'm using the exact same binaries as I was when I created the ticket. And I'm fairly certain that I was seeing a failure on Jenkins as well (which runs on Linux), but those job logs are no longer available.

So, um, I guess I'll close this issue. I'm very confused.

[edhartnett]

OK, thanks for doublechecking.

Let us know if you see this problem again.