Merge branch 'main' into fix/zarr-v3

* main:
  Fix multiple grouping with missing groups (pydata#9650)
  flox: Properly propagate multiindex (pydata#9649)
  Update Datatree html repr to indicate inheritance (pydata#9633)
  Re-implement map_over_datasets using group_subtrees (pydata#9636)
  fix zarr intersphinx (pydata#9652)
  Replace black and blackdoc with ruff-format (pydata#9506)
  Fix error and missing code cell in io.rst (pydata#9641)
  Support alternative names for the root node in DataTree.from_dict (pydata#9638)
  Updates to DataTree.equals and DataTree.identical (pydata#9627)
  DOC: Clarify error message in open_dataarray (pydata#9637)
  Add zip_subtrees for paired iteration over DataTrees (pydata#9623)
  Type check datatree tests (pydata#9632)
  Add missing `memo` argument to DataTree.__deepcopy__ (pydata#9631)
  Bug fixes for DataTree indexing and aggregation (pydata#9626)
  Add inherit=False option to DataTree.copy() (pydata#9628)
  docs(groupby): mention deprecation of `squeeze` kwarg (pydata#9625)
  Migration guide for users of old datatree repo (pydata#9598)
  Reimplement Datatree typed ops (pydata#9619)

.pre-commit-config.yaml

@@ -15,20 +15,15 @@ repos:
     # Ruff version.
     rev: 'v0.6.9'
     hooks:
+      - id: ruff-format
       - id: ruff
         args: ["--fix", "--show-fixes"]
-  # https://github.com/python/black#version-control-integration
-  - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 24.8.0
-    hooks:
-      - id: black-jupyter
   - repo: https://github.com/keewis/blackdoc
     rev: v0.3.9
     hooks:
       - id: blackdoc
         exclude: "generate_aggregations.py"
         additional_dependencies: ["black==24.8.0"]
-      - id: blackdoc-autoupdate-black
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: v1.11.2
     hooks:

CORE_TEAM_GUIDE.md

@@ -271,8 +271,7 @@ resources such as:
    [NumPy documentation guide](https://numpy.org/devdocs/dev/howto-docs.html#documentation-style)
    for docstring conventions.
 - [`pre-commit`](https://pre-commit.com) hooks for autoformatting.
-- [`black`](https://github.com/psf/black) autoformatting.
-- [`flake8`](https://github.com/PyCQA/flake8) linting.
+- [`ruff`](https://github.com/astral-sh/ruff) autoformatting and linting.
 - [python-xarray](https://stackoverflow.com/questions/tagged/python-xarray) on Stack Overflow.
 - [@xarray_dev](https://twitter.com/xarray_dev) on Twitter.
 - [xarray-dev](https://discord.gg/bsSGdwBn) discord community (normally only used for remote synchronous chat during sprints).

DATATREE_MIGRATION_GUIDE.md

@@ -0,0 +1,63 @@
+# Migration guide for users of `xarray-contrib/datatree`
+
+_15th October 2024_
+
+This guide is for previous users of the prototype `datatree.DataTree` class in the `xarray-contrib/datatree repository`. That repository has now been archived, and will not be maintained. This guide is intended to help smooth your transition to using the new, updated `xarray.DataTree` class.
+
+> [!IMPORTANT]
+> There are breaking changes! You should not expect that code written with `xarray-contrib/datatree` will work without any modifications.  At the absolute minimum you will need to change the top-level import statement, but there are other changes too.
+
+We have made various changes compared to the prototype version. These can be split into three categories: data model changes, which affect the hierarchal structure itself; integration with xarray's IO backends; and minor API changes, which mostly consist of renaming methods to be more self-consistent.
+
+### Data model changes
+
+The most important changes made are to the data model of `DataTree`. Whilst previously data in different nodes was unrelated and therefore unconstrained, now trees have "internal alignment" - meaning that dimensions and indexes in child nodes must exactly align with those in their parents.
+
+These alignment checks happen at tree construction time, meaning there are some netCDF4 files and zarr stores that could previously be opened as `datatree.DataTree` objects using `datatree.open_datatree`, but now cannot be opened as `xr.DataTree` objects using `xr.open_datatree`. For these cases we added a new opener function `xr.open_groups`, which returns a `dict[str, Dataset]`. This is intended as a fallback for tricky cases, where the idea is that you can still open the entire contents of the file using `open_groups`, edit the `Dataset` objects, then construct a valid tree from the edited dictionary using `DataTree.from_dict`.
+
+The alignment checks allowed us to add "Coordinate Inheritance", a much-requested feature where indexed coordinate variables are now "inherited" down to child nodes. This allows you to define common coordinates in a parent group that are then automatically available on every child node. The distinction between a locally-defined coordinate variables and an inherited coordinate that was defined on a parent node is reflected in the `DataTree.__repr__`. Generally if you prefer not to have these variables be inherited you can get more similar behaviour to the old `datatree` package by removing indexes from coordinates, as this prevents inheritance.
+
+Tree structure checks between multiple trees (i.e., `DataTree.isomorophic`) and pairing of nodes in arithmetic has also changed. Nodes are now matched (with `xarray.group_subtrees`) based on their relative paths, without regard to the order in which child nodes are defined.
+
+For further documentation see the page in the user guide on Hierarchical Data.
+
+### Integrated backends
+
+Previously `datatree.open_datatree` used a different codepath from `xarray.open_dataset`, and was hard-coded to only support opening netCDF files and Zarr stores.
+Now xarray's backend entrypoint system has been generalized to include `open_datatree` and the new `open_groups`.
+This means we can now extend other xarray backends to support `open_datatree`! If you are the maintainer of an xarray backend we encourage you to add support for `open_datatree` and `open_groups`!
+
+Additionally:
+- A `group` kwarg has been added to `open_datatree` for choosing which group in the file should become the root group of the created tree.
+- Various performance improvements have been made, which should help when opening netCDF files and Zarr stores with large numbers of groups.
+- We anticipate further performance improvements being possible for datatree IO.
+
+### API changes
+
+A number of other API changes have been made, which should only require minor modifications to your code:
+- The top-level import has changed, from `from datatree import DataTree, open_datatree` to `from xarray import DataTree, open_datatree`. Alternatively you can now just use the `import xarray as xr` namespace convention for everything datatree-related.
+- The `DataTree.ds` property has been changed to `DataTree.dataset`, though `DataTree.ds` remains as an alias for `DataTree.dataset`.
+- Similarly the `ds` kwarg in the `DataTree.__init__` constructor has been replaced by `dataset`, i.e. use `DataTree(dataset=)` instead of `DataTree(ds=...)`.
+- The method `DataTree.to_dataset()` still exists but now has different options for controlling which variables are present on the resulting `Dataset`, e.g. `inherit=True/False`.
+- `DataTree.copy()` also has a new `inherit` keyword argument for controlling whether or not coordinates defined on parents are copied (only relevant when copying a non-root node).
+- The `DataTree.parent` property is now read-only. To assign a ancestral relationships directly you must instead use the `.children` property on the parent node, which remains settable.
+- Similarly the `parent` kwarg has been removed from the `DataTree.__init__` constuctor.
+- DataTree objects passed to the `children` kwarg in `DataTree.__init__` are now shallow-copied.
+- `DataTree.as_array` has been replaced by `DataTree.to_dataarray`.
+- A number of methods which were not well tested have been (temporarily) disabled. In general we have tried to only keep things that are known to work, with the plan to increase API surface incrementally after release.
+
+## Thank you!
+
+Thank you for trying out `xarray-contrib/datatree`!
+
+We welcome contributions of any kind, including good ideas that never quite made it into the original datatree repository. Please also let us know if we have forgotten to mention a change that should have been listed in this guide.
+
+Sincerely, the datatree team:
+
+Tom Nicholas,
+Owen Littlejohns,
+Matt Savoie,
+Eni Awowale,
+Alfonso Ladino,
+Justus Magin,
+Stephan Hoyer

ci/min_deps_check.py

@@ -3,6 +3,7 @@
 publication date. Compare it against requirements/min-all-deps.yml to verify the
 policy on obsolete dependencies is being followed. Print a pretty report :)
 """
+
 from __future__ import annotations
 
 import itertools
@@ -16,7 +17,6 @@
 
 CHANNELS = ["conda-forge", "defaults"]
 IGNORE_DEPS = {
-    "black",
     "coveralls",
     "flake8",
     "hypothesis",

ci/requirements/all-but-dask.yml

@@ -3,7 +3,6 @@ channels:
   - conda-forge
   - nodefaults
 dependencies:
-  - black
   - aiobotocore
   - array-api-strict
   - boto3

doc/api.rst

@@ -749,6 +749,17 @@ Manipulate the contents of a single ``DataTree`` node.
    DataTree.assign
    DataTree.drop_nodes
 
+DataTree Operations
+-------------------
+
+Apply operations over multiple ``DataTree`` objects.
+
+.. autosummary::
+   :toctree: generated/
+
+   map_over_datasets
+   group_subtrees
+
 Comparisons
 -----------
 
@@ -849,20 +860,20 @@ Aggregate data in all nodes in the subtree simultaneously.
    DataTree.cumsum
    DataTree.cumprod
 
-.. ndarray methods
-.. ---------------
+ndarray methods
+---------------
 
-.. Methods copied from :py:class:`numpy.ndarray` objects, here applying to the data in all nodes in the subtree.
+Methods copied from :py:class:`numpy.ndarray` objects, here applying to the data in all nodes in the subtree.
 
-.. .. autosummary::
-..    :toctree: generated/
+.. autosummary::
+   :toctree: generated/
 
-..    DataTree.argsort
+   DataTree.argsort
+   DataTree.conj
+   DataTree.conjugate
+   DataTree.round
 ..    DataTree.astype
 ..    DataTree.clip
-..    DataTree.conj
-..    DataTree.conjugate
-..    DataTree.round
 ..    DataTree.rank
 
 .. Reshaping and reorganising
@@ -954,7 +965,6 @@ DataTree methods
 
    open_datatree
    open_groups
-   map_over_datasets
    DataTree.to_dict
    DataTree.to_netcdf
    DataTree.to_zarr

doc/conf.py

@@ -347,7 +347,7 @@
     "scipy": ("https://docs.scipy.org/doc/scipy", None),
     "sparse": ("https://sparse.pydata.org/en/latest/", None),
     "xarray-tutorial": ("https://tutorial.xarray.dev/", None),
-    "zarr": ("https://zarr.readthedocs.io/en/latest/", None),
+    "zarr": ("https://zarr.readthedocs.io/en/stable/", None),
 }
 
 

doc/contributing.rst

@@ -549,11 +549,7 @@ Code Formatting
 
 xarray uses several tools to ensure a consistent code format throughout the project:
 
-- `Black <https://black.readthedocs.io/en/stable/>`_ for standardized
-  code formatting,
-- `blackdoc <https://blackdoc.readthedocs.io/en/stable/>`_ for
-  standardized code formatting in documentation,
-- `ruff <https://github.com/charliermarsh/ruff/>`_ for code quality checks and standardized order in imports
+- `ruff <https://github.com/astral-sh/ruff>`_ for formatting, code quality checks and standardized order in imports
 - `absolufy-imports <https://github.com/MarcoGorelli/absolufy-imports>`_ for absolute instead of relative imports from different files,
 - `mypy <http://mypy-lang.org/>`_ for static type checking on `type hints
   <https://docs.python.org/3/library/typing.html>`_.
@@ -1069,7 +1065,7 @@ PR checklist
   - Test the code using `Pytest <http://doc.pytest.org/en/latest/>`_. Running all tests (type ``pytest`` in the root directory) takes a while, so feel free to only run the tests you think are needed based on your PR (example: ``pytest xarray/tests/test_dataarray.py``). CI will catch any failing tests.
   - By default, the upstream dev CI is disabled on pull request and push events. You can override this behavior per commit by adding a ``[test-upstream]`` tag to the first line of the commit message. For documentation-only commits, you can skip the CI per commit by adding a ``[skip-ci]`` tag to the first line of the commit message.
 
-- **Properly format your code** and verify that it passes the formatting guidelines set by `Black <https://black.readthedocs.io/en/stable/>`_ and `Flake8 <http://flake8.pycqa.org/en/latest/>`_. See `"Code formatting" <https://docs.xarray.dev/en/stablcontributing.html#code-formatting>`_. You can use `pre-commit <https://pre-commit.com/>`_ to run these automatically on each commit.
+- **Properly format your code** and verify that it passes the formatting guidelines set by `ruff <https://github.com/astral-sh/ruff>`_. See `"Code formatting" <https://docs.xarray.dev/en/stablcontributing.html#code-formatting>`_. You can use `pre-commit <https://pre-commit.com/>`_ to run these automatically on each commit.
 
   - Run ``pre-commit run --all-files`` in the root directory. This may modify some files. Confirm and commit any formatting changes.
 

doc/user-guide/hierarchical-data.rst

@@ -362,21 +362,26 @@ This returns an iterable of nodes, which yields them in depth-first order.
     for node in vertebrates.subtree:
         print(node.path)
 
-A very useful pattern is to use :py:class:`~xarray.DataTree.subtree` conjunction with the :py:class:`~xarray.DataTree.path` property to manipulate the nodes however you wish,
-then rebuild a new tree using :py:meth:`xarray.DataTree.from_dict()`.
+Similarly, :py:class:`~xarray.DataTree.subtree_with_keys` returns an iterable of
+relative paths and corresponding nodes.
 
+A very useful pattern is to iterate over :py:class:`~xarray.DataTree.subtree_with_keys`
+to manipulate nodes however you wish, then rebuild a new tree using
+:py:meth:`xarray.DataTree.from_dict()`.
 For example, we could keep only the nodes containing data by looping over all nodes,
 checking if they contain any data using :py:class:`~xarray.DataTree.has_data`,
 then rebuilding a new tree using only the paths of those nodes:
 
 .. ipython:: python
 
-    non_empty_nodes = {node.path: node.dataset for node in dt.subtree if node.has_data}
+    non_empty_nodes = {
+        path: node.dataset for path, node in dt.subtree_with_keys if node.has_data
+    }
     xr.DataTree.from_dict(non_empty_nodes)
 
 You can see this tree is similar to the ``dt`` object above, except that it is missing the empty nodes ``a/c`` and ``a/c/d``.
 
-(If you want to keep the name of the root node, you will need to add the ``name`` kwarg to :py:class:`~xarray.DataTree.from_dict`, i.e. ``DataTree.from_dict(non_empty_nodes, name=dt.root.name)``.)
+(If you want to keep the name of the root node, you will need to add the ``name`` kwarg to :py:class:`~xarray.DataTree.from_dict`, i.e. ``DataTree.from_dict(non_empty_nodes, name=dt.name)``.)
 
 .. _manipulating trees:
 
@@ -573,38 +578,78 @@ Then calculate the RMS value of these signals:
 
 .. _multiple trees:
 
-We can also use the :py:meth:`~xarray.map_over_datasets` decorator to promote a function which accepts datasets into one which
-accepts datatrees.
+We can also use :py:func:`~xarray.map_over_datasets` to apply a function over
+the data in multiple trees, by passing the trees as positional arguments.
 
 Operating on Multiple Trees
 ---------------------------
 
 The examples so far have involved mapping functions or methods over the nodes of a single tree,
 but we can generalize this to mapping functions over multiple trees at once.
 
+Iterating Over Multiple Trees
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To iterate over the corresponding nodes in multiple trees, use
+:py:func:`~xarray.group_subtrees` instead of
+:py:class:`~xarray.DataTree.subtree_with_keys`. This combines well with
+:py:meth:`xarray.DataTree.from_dict()` to build a new tree:
+
+.. ipython:: python
+
+    dt1 = xr.DataTree.from_dict({"a": xr.Dataset({"x": 1}), "b": xr.Dataset({"x": 2})})
+    dt2 = xr.DataTree.from_dict(
+        {"a": xr.Dataset({"x": 10}), "b": xr.Dataset({"x": 20})}
+    )
+    result = {}
+    for path, (node1, node2) in xr.group_subtrees(dt1, dt2):
+        result[path] = node1.dataset + node2.dataset
+    xr.DataTree.from_dict(result)
+
+Alternatively, you apply a function directly to paired datasets at every node
+using :py:func:`xarray.map_over_datasets`:
+
+.. ipython:: python
+
+    xr.map_over_datasets(lambda x, y: x + y, dt1, dt2)
+
 Comparing Trees for Isomorphism
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 For it to make sense to map a single non-unary function over the nodes of multiple trees at once,
-each tree needs to have the same structure. Specifically two trees can only be considered similar, or "isomorphic",
-if they have the same number of nodes, and each corresponding node has the same number of children.
-We can check if any two trees are isomorphic using the :py:meth:`~xarray.DataTree.isomorphic` method.
+each tree needs to have the same structure. Specifically two trees can only be considered similar,
+or "isomorphic", if the full paths to all of their descendent nodes are the same.
+
+Applying :py:func:`~xarray.group_subtrees` to trees with different structures
+raises :py:class:`~xarray.TreeIsomorphismError`:
 
 .. ipython:: python
     :okexcept:
 
-    dt1 = xr.DataTree.from_dict({"a": None, "a/b": None})
-    dt2 = xr.DataTree.from_dict({"a": None})
-    dt1.isomorphic(dt2)
+    tree = xr.DataTree.from_dict({"a": None, "a/b": None, "a/c": None})
+    simple_tree = xr.DataTree.from_dict({"a": None})
+    for _ in xr.group_subtrees(tree, simple_tree):
+        ...
+
+We can explicitly also check if any two trees are isomorphic using the :py:meth:`~xarray.DataTree.isomorphic` method:
+
+.. ipython:: python
+
+    tree.isomorphic(simple_tree)
 
-    dt3 = xr.DataTree.from_dict({"a": None, "b": None})
-    dt1.isomorphic(dt3)
+Corresponding tree nodes do not need to have the same data in order to be considered isomorphic:
 
-    dt4 = xr.DataTree.from_dict({"A": None, "A/B": xr.Dataset({"foo": 1})})
-    dt1.isomorphic(dt4)
+.. ipython:: python
+
+    tree_with_data = xr.DataTree.from_dict({"a": xr.Dataset({"foo": 1})})
+    simple_tree.isomorphic(tree_with_data)
+
+They also do not need to define child nodes in the same order:
+
+.. ipython:: python
 
-If the trees are not isomorphic a :py:class:`~xarray.TreeIsomorphismError` will be raised.
-Notice that corresponding tree nodes do not need to have the same name or contain the same data in order to be considered isomorphic.
+    reordered_tree = xr.DataTree.from_dict({"a": None, "a/c": None, "a/b": None})
+    tree.isomorphic(reordered_tree)
 
 Arithmetic Between Multiple Trees
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/user-guide/io.rst

@@ -19,16 +19,11 @@ format (recommended).
 
     np.random.seed(123456)
 
-You can `read different types of files <https://docs.xarray.dev/en/stable/user-guide/io.html>`_
-in `xr.open_dataset` by specifying the engine to be used:
+You can read different types of files in `xr.open_dataset` by specifying the engine to be used:
 
-.. ipython:: python
-    :okexcept:
-    :suppress:
-
-    import xarray as xr
+.. code:: python
 
-    xr.open_dataset("my_file.grib", engine="cfgrib")
+    xr.open_dataset("example.nc", engine="netcdf4")
 
 The "engine" provides a set of instructions that tells xarray how
 to read the data and pack them into a `dataset` (or `dataarray`).

doc/whats-new.rst

@@ -23,13 +23,15 @@ New Features
 ~~~~~~~~~~~~
 - ``DataTree`` related functionality is now exposed in the main ``xarray`` public
   API. This includes: ``xarray.DataTree``, ``xarray.open_datatree``, ``xarray.open_groups``,
-  ``xarray.map_over_datasets``, ``xarray.register_datatree_accessor`` and
-  ``xarray.testing.assert_isomorphic``.
+  ``xarray.map_over_datasets``, ``xarray.group_subtrees``,
+  ``xarray.register_datatree_accessor`` and ``xarray.testing.assert_isomorphic``.
   By `Owen Littlejohns <https://github.com/owenlittlejohns>`_,
   `Eni Awowale <https://github.com/eni-awowale>`_,
   `Matt Savoie <https://github.com/flamingbear>`_,
   `Stephan Hoyer <https://github.com/shoyer>`_ and
   `Tom Nicholas <https://github.com/TomNicholas>`_.
+- A migration guide for users of the prototype `xarray-contrib/datatree repository <https://github.com/xarray-contrib/datatree>`_ has been added, and can be found in the `DATATREE_MIGRATION_GUIDE.md` file in the repository root.
+  By `Tom Nicholas <https://github.com/TomNicholas>`_.
 - Added zarr backends for :py:func:`open_groups` (:issue:`9430`, :pull:`9469`).
   By `Eni Awowale <https://github.com/eni-awowale>`_.
 - Added support for vectorized interpolation using additional interpolators
@@ -65,11 +67,13 @@ Bug fixes
   the non-missing times could in theory be encoded with integers
   (:issue:`9488`, :pull:`9497`). By `Spencer Clark
   <https://github.com/spencerkclark>`_.
-- Fix a few bugs affecting groupby reductions with `flox`. (:issue:`8090`, :issue:`9398`).
+- Fix a few bugs affecting groupby reductions with `flox`. (:issue:`8090`, :issue:`9398`, :issue:`9648`).
   By `Deepak Cherian <https://github.com/dcherian>`_.
 - Fix the safe_chunks validation option on the to_zarr method
   (:issue:`5511`, :pull:`9559`). By `Joseph Nowak
   <https://github.com/josephnowak>`_.
+- Fix binning by multiple variables where some bins have no observations. (:issue:`9630`).
+  By `Deepak Cherian <https://github.com/dcherian>`_.
 
 Documentation
 ~~~~~~~~~~~~~