support chunks in open_groups and open_datatree
#9660
Conversation
TomNicholas
added
topic-backends
topic-DataTree
|
Thanks for doing this @keewis! |
xarray/backends/api.py
Outdated
| } | ||
| ) | ||
|
|
||
| # ds.set_close(backend_ds._close) |
There was a problem hiding this comment.
backend_tree should have been created using datatree_from_dict_with_io_cleanup, so one way to handle this could be just to copy over the _close attribute from every node of backend_tree?
There was a problem hiding this comment.
the question is, do we even need that here? I copied this from open_dataset where this is explicitly set, but since datatree_from_dict_with_io_cleanup does this already we might be able to just remove it?
The only reason why I kept the commented-out line is to discuss whether the shift in paradigm (have the backend set _close vs. do it for all backends the same way) is intentional, and if we should do the same for open_dataset.
There was a problem hiding this comment.
I agree it would be nice to remove this, I'm just worried that mapping over the each .dataset might not properly propagate ._close (does it? should it?)
There was a problem hiding this comment.
It does not (I think), so I'm explicitly copying it over. So far that doesn't appear to cause anything to break.
|
I've copied over the docstring of |
Co-authored-by: Tom Nicholas <tom@cworthy.org>
Co-authored-by: Tom Nicholas <tom@cworthy.org>
Co-authored-by: Tom Nicholas <TomNicholas@users.noreply.github.com>
|
I'm happy to merge this @keewis ? |
me too! Feel free to go ahead and merge. |
* adding draft for fixing behaviour for group parameter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * new trial * new trial * fixing duplicate pahts and path in the root group * removing yield str(gpath) * implementing the proposed solution to hdf5 and netcdf backends * adding changes to whats-new.rst * removing encoding['source_group'] line to avoid conflicts with PR #9660 * adding test * adding test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding assert subgroup_tree.root.parent is None * modifying tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update xarray/tests/test_backends_datatree.py Co-authored-by: Justus Magin <keewis@users.noreply.github.com> * applying suggested changes * updating test * adding Justus and Alfonso to the list of contributors to the DataTree entry * adding Justus and Alfonso to the list of contributors to the DataTree entry --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Tom Nicholas <tom@cworthy.org> Co-authored-by: Justus Magin <keewis@users.noreply.github.com>
In trying to support
chunksthe way we do foropen_datasetI had to add a lot of parameters to the top-levelopen_datatreeandopen_groupsfunctions.I'm also still looking for the equivalent of
_protect_dataset_variables_inplace, and finally_dataset_from_backend_datasethas been the place to callset_close, while #9651 pushed this to the backends for datatree.No tests yet, and I also want to improve the docstrings before merging.
(the chunked array methods –
chunk,load,compute, andpersist– should be a separate PR)cc @TomNicholas, @shoyer