Saving 30Gb Zarr to NetCDF

[mliukis]

I am having troubles with converting 30Gb large Zarr store to NetCDF with xarray. It appears that each data array of the dataset is read in as a whole into memory and the process fails with:

numpy.core._exceptions.MemoryError: Unable to allocate 74.3 GiB for an array with shape (28658, 834, 834) and data type float32
Is there a way to do the conversion in chunks? Or are there any plans of such support for large data sets?

[shoyer]

Streaming writes to netCDF are supported if you open the Zarr store with Dask. Take a look at http://xarray.pydata.org/en/stable/user-guide/dask.html to get started.

[mliukis]

Thank you so much! That is a saver :)

[mliukis]

@shoyer Just FYI that the following version of Dask does not work with streaming writes to NetCDF:
dask 2021.7.1 pyhd8ed1ab_0 conda-forge

I spent a couple of hours figuring out why conversion to NetCDF of Zarr stores open with Dask (Version 2.30.0) were working on my Mac laptop, while exact same conversion was running out of memory when ran within Docker container with exact same amount of RAM as my laptop. The conda was installing Dask 2021.7.1 version when building Docker image environment. Something goes wrong as you can observe memory just going up until it reaches the maximum available, then the process is killed.