Memory allocation error when exporting sliced xarray data array to a numpy array

[lassiterdc]

The sliced data array is 0.002 megabytes but when I try to export it to a numpy array, xarray tries to load the entire unsliced data array into memory which causes a memory allocation error (which is in itself odd since the dataset is only 318MB).

MRMS precipitation data in grib2 format (318MB): https://www.dropbox.com/s/xq0vvamcydu3qn2/PrecipRate_20210524.grib2?dl=0

# you also need cfgrib installed
# conda install -c conda-forge cfgrib
# https://github.com/ecmwf/cfgrib

#%% Data processing
import xarray as xr

f_in_grib = "PrecipRate_20210524.grib2"

mrms = xr.open_dataset(f_in_grib, engine="cfgrib", chunks = {'latitude':10, 'longitude':10})

idx = dict(latitude=1808, longitude=5374)

data_at_gages = mrms[idx]

#%% inspection
print("The dataset 'data_at_gages' is {} megabytes.".format(data_at_gages.unknown.nbytes/1e6))
# The dataset 'data_at_gages' is 0.0021 megabytes.

print("The dimensions of the sliced data array are: {}".format(data_at_gages.unknown.shape))
# The dimensions of the sliced data array are: (525,)

#%% error
data_at_gages.unknown.values.to_numpy

# MemoryError: Unable to allocate 47.9 GiB for an array with shape (525, 3500, 7000) and data type float32

[mliukis]

It could be the Dask issue - please take a look at my other reported issue. Downgrading Dask took care of the problem for me.

[lassiterdc]

Ah you are right, it just keeps incrementing up till I cap my memory... Is there a way to avoid that or do I just need tons of RAM to access a time series for a single lat/long?

[mliukis]

That what was happening to my process converting huge Zarr store to NetCDF - the memory usage kept climbing though I was using chunked Dask arrays in xarray.
Sorry for not being of much help. From my experience with xarray, it could be the xarray and dependent Python packages interaction that is causing the problem. My other xarray debugging experience was that I had to downgrade Zarr package as it was causing corrupted datetime objects in Zarr. Sigh...
I hope some xarray gurus will come to your rescue :) Good luck!

[mliukis]

Something I just noticed in the following line:

#%% error data_at_gages.unknown.values.to_numpy

Shouldn't data_at_gages.unknown.values be already of the np.ndarray type according to values? Then you don't need to call to_numpy, and I would expect you getting a syntax error instead of running out of memory :)
Most likely it's running out of memory accessing values

[lassiterdc]

Ah okay so it sounds like that memory usage climbing is evidence of some kind of malfunction so I'll keep at it.

Your second comment gave me a glimmer of hope but I tried running data_at_gages.to_netcdf() and ran into the same issue. If it means anything, I was able to run data_at_gages.to_dask_dataframe() but any operation causes the same memory usage climb it seems.

[mliukis]

Makes me think it's a Dask version issue. Did you try to downgrade Dask to the same version that worked for me:

dask                      2.30.0                     py_0    conda-forge

?