Use numbagg for ffill by default

[max-sixty]

The main perf advantage here is the array doesn't need to be unstacked & stacked, which is a huge win for large multi-dimensional arrays... (I actually was hitting a memory issue running an ffill on my own, and so thought I'd get this done!)

We could move these methods to DataWithCoords, since they're almost the same implementation between a DataArray & Dataset, and exactly the same for numbagg's implementation

---

For transparency — the logic of "check for numbagg, check for bottleneck" I wouldn't rate at my most confident. But I'm more confident that just installing numbagg will work. And if that works well enough, we could consider only supporting numbagg for some of these in the future.

I also haven't done the benchmarks here — though the functions are relatively well benchmarked at numbagg. I'm somewhat trading off getting through these (rolling functions are coming up too) vs. doing fewer slower, and leaning towards the former, but feedback welcome...

• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[dcherian]

the array doesn't need to be unstacked & stacked

Where does this happen?

---

We should be able to switch between bottleneck and numbagg here:

xarray/xarray/core/duck_array_ops.py

Line 691 in f63ede9

def push(array, n, axis):

and here:

xarray/xarray/core/dask_array_ops.py

Line 58 in f63ede9

def push(array, n, axis):

then you'll get dask-aware ffill for free :)

[max-sixty]

the array doesn't need to be unstacked & stacked

Where does this happen?

Super weird — I have a stack trace from memray that has:

da.ffill("t")
ds = self._to_temp_dataset().unstack(dim, fill_value, sparse)
result = result._unstack_once(d, stacked_indexes[d], fill_value, sparse) 

It's a sampling profiler, so it misses frames. But after 10 mins of looking, I can't see how that would possibly happen — I can't see where this would get called.

Though IIRC bottleneck doesn't deal well with arrays with lots of dimensions, and that array has lots of dimensions; so I think it's still possible that something is unstacking to reduce the dimensions even I can't find it atm.

Anyway!!

---

We should be able to switch between bottleneck and numbagg here:

OK great!

[max-sixty]

I refactored this to use the duck_array_ops, that's much better. Regretfully all the previous code on this PR was just a worse version of that and was discarded.

I need to work through the version checks...

[max-sixty]

Unfortunately I'm seeing some segfaults when running with:

• dask
• target="parallel", which is the default for numbagg functions

This would not be a great experience. So I'll look at these before we consider merging; I suspect it'll be something upstream...

[max-sixty]

OK, this is all done I think:

• Using duck array module, since

  xarray/xarray/namedarray/utils.py

  Line 42 in cfe4d71

  def module_available(module: str) -> bool:

  doesn't get the version. (Open to anyone making changes there, I don't think it's perfect, and it's now oddly named, though I probably won't be the person to make more refinements there...)
• numbagg is fixed to switch to target="cpu" when running in multithreading
• I also raised that issue with numba

[dcherian]

Just some nits

[dcherian]

shall we boost min supported numbagg version?

[max-sixty]

I think we basically keep the 12 month cycle — it's some good infra, and I would like to be a good citizen for that process rather than deviate to save myself some boilerplate...

[dcherian]

what are "wide" arrays?

[max-sixty]

What's a better term? I agree it's not great now. A "wide" table has lots of column that it can parallelize along. i.e. the benchmarks at https://github.com/numbagg/numbagg/#nd are 5x better than bottleneck, but just above, on a single column, it's identical, because the benefit comes from the parallelism.