Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KACE-1-0-G fails clean-cmip6 with chunking issue in xclim_convert_360day_calendar_interpolate function #436

Closed
dgergel opened this issue Dec 9, 2021 · 1 comment
Assignees
Labels
Milestone

Comments

@dgergel
Copy link
Member

dgergel commented Dec 9, 2021

error from the logs:

INFO:dodola.services:Starting dodola service clean_cmip6 with args=('gs://scratch-170cd6ec/1acf3ca2-9f7a-4a3b-b842-8f4562c6e531/clean-cmip6-dev-vbgln-2364147584/timesliced.zarr'
, 'gs://scratch-170cd6ec/1acf3ca2-9f7a-4a3b-b842-8f4562c6e531/clean-cmip6-dev-vbgln-1245057932/standardized.zarr', True), kwargs={})
INFO:dodola.repository:Read gs://scratch-170cd6ec/1acf3ca2-9f7a-4a3b-b842-8f4562c6e531/clean-cmip6-dev-vbgln-2364147584/timesliced.zarr
Traceback (most recent call last):
  File "/opt/conda/bin/dodola", line 33, in <module>
    sys.exit(load_entry_point('dodola', 'console_scripts', 'dodola')())
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/opt/dodola/dodola/cli.py", line 477, in cleancmip6
    services.clean_cmip6(x, out, drop_leapdays)
  File "/opt/dodola/dodola/services.py", line 35, in service_logger
    func(*args, **kwargs)
  File "/opt/dodola/dodola/services.py", line 721, in clean_cmip6
    cleaned_ds = standardize_gcm(ds, leapday_removal)
  File "/opt/dodola/dodola/core.py", line 524, in standardize_gcm
    ds_converted = xclim_convert_360day_calendar_interpolate(
  File "/opt/dodola/dodola/core.py", line 647, in xclim_convert_360day_calendar_interpolate
    ds_out = ds_converted.interpolate_na("time", interpolation)
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/dataset.py", line 4875, in interpolate_na
    new = _apply_over_vars_with_dim(
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/missing.py", line 204, in _apply_over_vars_with_dim
    ds[name] = func(var, dim=dim, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/missing.py", line 349, in interp_na
    arr = apply_ufunc(
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/computation.py", line 1174, in apply_ufunc
    return apply_dataarray_vfunc(
File "/opt/conda/lib/python3.9/site-packages/xarray/core/computation.py", line 293, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/opt/conda/lib/python3.9/site-packages/xarray/core/computation.py", line 681, in apply_variable_ufunc
    raise ValueError(
ValueError: dimension time on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk int
o a single dask array chunk along this dimension, i.e., ``.chunk(dict(time=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantl
y increase memory usage.
@brews
Copy link
Member

brews commented Dec 9, 2021

I think we got through the rechunking issue with ClimateImpactLab/dodola#151.

Need to rerun and check. If there is a new issue, please close this and open a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants