Add open_virtual_mfdataset

[TomNicholas]

Here I have copied the code from xr.open_mfdataset, changed it to use open_virtual_dataset, and added an option to parallelize with lithops as an alternative to using dask.delayed.

I haven't even tried to run this yet, but I think this is the right approach @tomwhite? I realised we don't need cubed's blockwise because xarray.open_mfdataset has internal logic to turn the N-dimensional concat into a 1D map already, so lithops.map should be fine?

Also I think based on our conversation we should be able to use lithops.map instead of lithops.map_reduce like @thodson-usgs did in #203 because the tiny size of the virtual datasets being returned to the client means that we should be able to get away with a single reduction step on the client even at large scale? (see also #104 for justification that we only need to send back kB-sized objects).

• Adds open_virtual_mfdataset as suggested in open_virtual_mfdataset #345, but also sketches out how we might close both Trying to run open_virtual_dataset in parallel #95 and Serverless parallelization of reference generation #123
• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst
• New functions/methods are listed in api.rst
• New functionality has documentation

[TomNicholas]

This is the equivalent of @thodson-usgs 's map_references function

VirtualiZarr/examples/virtualizarr-with-lithops/virtualizarr-with-lithops.py

Line 25 in 1dbd119

def map_references(fil):

[TomNicholas]

IIUC this will cause every serverless worker to send a small virtual dataset back to the client process over the internet somehow

[TomNicholas]

This is only going to work if we have used loadable_variables to create indexes for 1D coordinates, so it's a good reason to implement the suggestion in #335 (comment)

[TomNicholas]

I don't like importing these deep xarray internals like this (though _infer_concat_order_from_positions and _nested_combine haven't changed since I wrote them 6 years ago), but the only alternative would be to make a general virtualizarr backend engine for xarray (see #35).

[TomNicholas]

I think the code would be more straightforward if the parallel primitive we used for lithops was the same as the one we used for dask.

[TomNicholas]

I believe all of this could also be useful upstream in xr.open_mfdataset

[tomwhite]

I realised we don't need cubed's blockwise because xarray.open_mfdataset has internal logic to turn the N-dimensional concat into a 1D map already, so lithops.map should be fine?

Yes, that should work fine. We may want to loosen/generalize blockwise slightly in Cubed to return an arbitrary object so it can be done with Cubed - but that can be done later.

Also I think based on our conversation we should be able to use lithops.map instead of lithops.map_reduce like @thodson-usgs did in #203 because the tiny size of the virtual datasets being returned to the client means that we should be able to get away with a single reduction step on the client even at large scale? (see also #104 for justification that we only need to send back kB-sized objects).

Agreed - it will be interesting to see this for large datasets. (It's also similar to the approach I've taken for storing data in Icechunk where the changesets are returned to the client - again, small kB-sized UUIDs.)

[TomNicholas]

I'm stuck on why this assert_identical on the datasets doesn't pass. It complains that all the coordinates are different, but every attribute I can think to check looks identical 😖

[TomNicholas]

Note to self: the docs and especially the readme should be rewritten to put this function front and center.