Initial release checklist

[TomNicholas]

What's here so far is a work-in-progress. I will consider this issue closed once the example in the Readme runs successfully.

For this we need:

• Definition of a ChunkManifest
• Definition of a ManifestArray
• Concatenation of ManifestArrays through merging their manifests
• Creation of ManifestArrays from kerchunk reference dicts
• ds_from_kerchunk_reference_dict function
• open_dataset_via_kerchunk function to read byte ranges and translate them to ManifestArray objects
• Correct handling of concatenation in xarray (Support xarray concat, including broadcasting #34)
• Option to avoid creating pesky indexes (see Required changes in xarray to avoid creating indexes #14)
• Xarray accessor to reconstruct kerchunk reference dicts from merged manifests (Xarray accessor to create kerchunk reference dict #28)
• Serialize the dicts to disk as json

This list would get us the ability to do this:

from virtualizarr import open_dataset_via_kerchunk

datasets = [open_dataset_via_kerchunk(filepath, no_indexes=True) for filepath in files]

combined = xr.combine_nested(datasets)

combined.virtualize.to_kerchunk(format='json')

# now can read using kerchunk mapper like normal

For the initial version we do not need:

• Xarray backend entrypoint plugin (this is just syntactic sugar) (Opening via xarray backendentrypoint #35)
• Automatic inferring of concatenation order (see Inferring concatenation order from coordinate data values #18)
• Writing out to parquet (Writing to parquet (following kerchunk format) #72)
• Automatic detection of different file formats (see Automatically detect different legacy file formats #10)
• Anything that requires zarr development, e.g. v3 support, variable-length chunking, concatenation of different codecs

Without this latter list we notably can't do combine_by_coords and we can't do open_dataset(filepath, engine='virtualizarr').

[TomNicholas]

(I just updated this to be much clearer about what the "MVP" involves)

[TomNicholas]

I'm ready to release this, but @MSanKeys963 says we can move this repo into the zarr-developers github org, and it would be great to do that move first 🙏

EDIT: See https://github.com/orgs/zarr-developers/discussions/70 for discussion on this

[jbusecke]

Adding a +1 here to move and release. I am working with @sashakames to prototype virtualizarr usage within ESGF and It would be great if we could pin versions.

[TomNicholas]

It's been moved!

[TomNicholas]

Now we've released v1.0 on PyPI and conda-forge with all the above initial features. The later features are all tracked by various issues.