Background no longer transparent

[jbednar]

Datashader 0.13 and master both show a background from the colormap, rather than letting the page background show through, in certain cases that used to work fine. E.g. the current Datashader docs show:

Here the background should have been (and used to be) NaN or zero in a way that renders to an alpha value of zero in the background, letting the page show through.

[ianthomas23]

This might be a red herring, but that looks suspiciously like the output from my currently open PR. Is there any way the docs build could be using the wrong branch?

[hoxbro]

The blueness was not present in the documentation for version 0.7.0 but it is there in version 0.11.1.

The sum returns an int64 if I change the type to uint32 the blueness disappears.

This could be related to some changes made on these lines:

datashader/datashader/transfer_functions/__init__.py

Lines 218 to 226 in 11a1924

	# Compute mask
	if np.issubdtype(data.dtype, np.bool_):
	mask = ~data
	data = data.astype(np.int8)
	else:
	if data.dtype.kind == 'u':
	mask = data == 0
	else:
	mask = np.isnan(data)

A guess would be this PR #910.

[ianthomas23]

Following on from @hoxbro's comment, if you change the line

        if data.dtype.kind == 'u':

to

        if data.dtype.kind in ('u', 'i'):

then this renders correctly and pytest runs OK. I don't know if there are wider ramifications of this change though.

[hoxbro]

Based on #910 (comment), I will say it was chosen to work this way - if this is still the case, I don't know.

If it is still the case, I suggest adding an argument to tf.shade like zero_as_nan, which can switch on the desired behavior.

[jbednar]

It was definitely not intended that the background ever show up blue like that. :-) So that's an unintentional implication of an intentional change. For floats, we are using NaN to represent missing data, i.e., the absence of any values. For an integer count, 0 is the absence of values, so 0 as NaN makes sense for count. But for a sum of integers, there could be negative numbers where the result is zero even though there is data, and in that case we can't treat 0 as no data. So I don't think it's safe to do if data.dtype.kind in ('u', 'i'):. But if we are summing values that we know are nonnegative, as in the example shown, then 0 as NaN should be safe.

The two cases where I can see this behavior in the docs are:

1. .sum() (tf.shade(merged.sum()) in Timeseries.ipynb and shown above): .sum() returns a signed integer array even though all its arguments are nonnegative. I think this is an xarray sum method, and thus nothing we could change. We could just update the docs to force the output to be unsigned and explain why. Alternatively, we could have a sum function that checks for uint inputs and promotes them to floats with zeros as NaNs before summing, but then people would have to know to use that, or we'd have to return an xarray subclass with sum redefined, which seems like overkill.
2. .where() (d3+d5 where d3==d5 in Pipeline.ipynb): seems similar; presumably .where also either returns a float without converting 0s to NaNs, or returns a signed integer output even for unsigned input.

Maybe best just to fix it in the docs?

[jbednar]

Yes, .where() returns float output for integer input:

That seems like a bug in xarray; I don't see why masking or summing should change the datatype. So I'll document that and just fix it at the docs level; I don't see anything different that Datashader should be doing here.