Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update ValueError for open_mfdataset for wild-card matching #5838

Closed
raybellwaves opened this issue Oct 4, 2021 · 0 comments · Fixed by #5851
Closed

update ValueError for open_mfdataset for wild-card matching #5838

raybellwaves opened this issue Oct 4, 2021 · 0 comments · Fixed by #5851

Comments

@raybellwaves
Copy link
Contributor

raybellwaves commented Oct 4, 2021

What happened:

Took engine="zarr" out of my open_mfdataset call and got

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_24527/4212238570.py in <module>
      3 TqdmCallback(desc="dask tasks").register()
      4 
----> 5 ds = xr.open_mfdataset(
      6     "s3://era5-pds/zarr/*/*/data/air_temperature_at_2_metres.zarr",
      7     parallel=True,

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combine_attrs, **kwargs)
    860             paths = [fs.get_mapper(path) for path in paths]
    861         elif is_remote_uri(paths):
--> 862             raise ValueError(
    863                 "cannot do wild-card matching for paths that are remote URLs: "
    864                 "{!r}. Instead, supply paths as an explicit list of strings.".format(

ValueError: cannot do wild-card matching for paths that are remote URLs: 's3://era5-pds/zarr/*/*/data/air_temperature_at_2_metres.zarr'. Instead, supply paths as an explicit list of strings.

What you expected to happen:

Give a suggestion that this can be fixed if engine="zarr"

Minimal Complete Verifiable Example:

import xarray as xr

ds = xr.open_mfdataset(
    "s3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr",
    backend_kwargs=dict(storage_options={"anon": True}),
)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ray/miniconda3/envs/main/lib/python3.9/site-packages/xarray/backends/api.py", line 862, in open_mfdataset
    raise ValueError(
ValueError: cannot do wild-card matching for paths that are remote URLs: 's3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr'. Instead, supply paths as an explicit list of strings.
ds = xr.open_mfdataset(
    "s3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr",
    backend_kwargs=dict(storage_options={"anon": True}),
    engine="zarr",
)
# works

Anything else we need to know?:

message here: https://github.com/pydata/xarray/blob/main/xarray/backends/api.py#L861

Environment:

Output of xr.show_versions()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant