ds.notnull() fails with AttributeError on pandas 0.21.0rc1

[gerritholl]

xarray.Dataset({"A": ("x", arange(5))}).notnull() fails with an 'AttributeError' when using numpy 1.13.3, xarray 0.9.6, and pandas 0.21.0rc1. The AttributeError is raised by pandas; see below.

$ cat mwe.py
#!/usr/bin/env python3.6

import numpy
print(numpy.__version__)
import xarray
print(xarray.__version__)
import pandas
print(pandas.__version__)

xarray.Dataset({"A": ("x", numpy.arange(5))}).notnull()
$ ./mwe.py
1.13.3
0.9.6
0.21.0rc1
Traceback (most recent call last):
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/ops.py", line 193, in func
    return getattr(self, name)(*args, **kwargs)
AttributeError: 'Variable' object has no attribute 'notna'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./mwe.py", line 10, in <module>
    xarray.Dataset({"A": ("x", numpy.arange(5))}).notnull()
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/dataset.py", line 2485, in func
    ds._variables[k] = f(self._variables[k], *args, **kwargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/xarray/core/ops.py", line 195, in func
    return f(self, *args, **kwargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 212, in notna
    res = isna(obj)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 45, in isna
    return _isna(obj)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/pandas/core/dtypes/missing.py", line 60, in _isna_new
    return obj._constructor(obj._data.isna(func=isna))
AttributeError: 'Variable' object has no attribute '_constructor'

[gerritholl]

Just confirmed this is caused by a change in pandas somewhere between 0.20.3 and 0.21.0rc1. I don't know if that is a bug in pandas, or a deliberate change that xarray will somehow need to handle, in particular after 0.21.0 final is released.

[gerritholl]

The offending commit is pandas-dev/pandas@7930202

[gerritholl]

What I still don't know: is this a bug in xarray or a bug in pandas? Or neither?

[gerritholl]

I think we'd need to change PANDAS_UNARY_FUNCTIONS = ['isnull', 'notnull'] in ops.py; I'll have a try.

[gfyoung]

@gerritholl : Weird that that commit would have caused things to break, since the changes should have been backwards-compatible (all we did was alias).

[gerritholl]

I'm not sure if I understand correctly, but it appears xarray has a hardcoded list of names of pandas functions/methods that need to be treated in a particular way. I might be on the wrong track though.

[shoyer]

We actually dealt with this back in #1549, but need to get a new xarray release out promptly so users can upgrade pandas.

[gerritholl]

Oh, I missed that. I should have tried with xarray master.