You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
import s3fs
import xarray as xr
s3 = s3fs.S3FileSystem(anon=True)
zarr_store = s3fs.S3Map(root=cube_url, s3=s3_in, check=False)
with xr.open_dataset(zarr_store, decode_timedelta=False, engine='zarr', consolidated=True, chunks={'mid_date': 250}) as ds:
...
I am getting the following user warning:
..../site-packages/xarray/core/dataset.py:410: UserWarning: Specified Dask chunks (250, 250, 250, 250, 250, 194) would separate on disks chunk shape 361 for dimension mid_date. This could degrade performance. Consider rechunking after loading instead.
I don't get this warning when I open the same Zarr store from the local file system. Does anybody know why I am getting this warning when Zarr is stored in S3 bucket?
I need to make sure that I open existing Zarr with Dask as I need to stream write it to the NetCDF file later on in the processing.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
When reading Zarr store from S3 bucket:
I am getting the following user warning:
..../site-packages/xarray/core/dataset.py:410: UserWarning: Specified Dask chunks (250, 250, 250, 250, 250, 194) would separate on disks chunk shape 361 for dimension mid_date. This could degrade performance. Consider rechunking after loading instead.
I don't get this warning when I open the same Zarr store from the local file system. Does anybody know why I am getting this warning when Zarr is stored in S3 bucket?
I need to make sure that I open existing Zarr with Dask as I need to stream write it to the NetCDF file later on in the processing.
Beta Was this translation helpful? Give feedback.
All reactions