Support for dask arrays

[bouweandela]

Because the datasets we use with the ESMValTool project are typically too large to fit into memory, we need support for lazy vertical interpolation in the ESMValCore package, see also ESMValGroup/ESMValCore#35.

Would support for dask arrays be a feature that is interesting for python-stratify? I think it could be used by a wider target audience if it's implemented here than if we only implement it for our own software. If there is an interest, I could make a pull request to add support for dask arrays to this package?

[bjlittle]

@bouweandela Sorry I'm late to the party, but if this is still of value to you guys and have capacity, then please do extend python-stratify to cope with dask arrays... that would be a definite community win for sure 👍

Awesome, thanks!

[bjlittle]

@bouweandela Could you elaborate on your use case at ESMValTool so that I can better understand what you need?

I've got some thoughts on how to go about this, but if you could share data, example connective tissue code, metrics on bottlenecks et al, then that would be super useful.
Thanks 👍

[bouweandela]

@bjlittle For the ESMValCore we need to be able to deal with input data that is larger than the amount of available RAM and we would like to use iris functions as much as possible, because of the limited amount of programming experts available in the project. We discussed the issue some time ago when you were at the workshop in Bavaria, resulting in this project board card. It contains the links to the relevant code too.

I had a go at changing python-stratify interpolator for the linear and nearest schemes some time ago using the same strategy I used for making regridding lazy in SciTools/iris#3701, but ran into some performance issues. I can share the code if this is useful?

I noticed we do not currently have support for the conservative scheme in ESMValCore and so far no-one has asked about it, but maybe it would be relevant. @zklaus Would you have an opinion on this?

You mentioned that you think a complete rewrite of the python-stratify package with numba would be a better idea, could you elaborate on that?

About the source and target levels we use in our data: this is CMIP data, so typically the pressure levels are a 1D array (time and lat/lon independent), though I'm not entirely sure if this is also the case for all ocean data. @zklaus Would you know more about this?

[zklaus]

On the conservative regridding: Just to be clear, for the horizontal regridding this definitively in demand and is also supported, at least for irregular grids. For vertical regridding, I am not so sure. Actually @axel-lauer may have a more qualified opinion on that from his cloud perspective.

Generally, CMIP is wide and almost anything happens. Most ocean models use simple depth as their z coordinate, but have a look at ESMValGroup/ESMValCore#895 for an example of an ocean model that uses density as its z-coordinate and delivers depth information in two separate variables, zhalf and zfull, time-dependent, of course.

[axel-lauer]

While it is true that many atmospheric 3-dim variables in CMIP are stored on pressure levels that are constant in time, there are also variables that are stored on hybrid levels. For example cloud variables such as 3-dim cloud cover (cl), cloud liquid water content (clw) or cloud ice content (cli) are stored on hybrid levels. Such hybrid levels typically use time-dependent reference fields (e.g. surface pressure) for the definition of the vertical levels, which makes the source levels for vertical interpolation also time-dependent. In my opinion, this would make lazy treatment of vertical interpolation also very intersting.

Given how interpolation is currently typically used, I would see the main priority on horizontal interpolation, and vertical interpolation as a secondary (but yet intersting and helpful!) aim.

[zklaus]

Good points, @axel-lauer! I wonder if true 3d regridding would be the gold standard here?