combine_by_coords() unexpectedly fails due to non-monotonic global indexes

[jrmagers]

I have a collection of datasets in which I think should be successfully combined into a single dataset using combine_by_coords(). However, I am getting the following ValueError:

ValueError: Resulting object does not have monotonic global indexes along dimension ref_level

Here is the code to reproduce the problem:

import xarray as xr
import pathlib

files = pathlib.Path('.').glob('*.nc')

data = list()
for f in files:
    data.append(xr.open_dataset(f))
    
combined_data = xr.combine_by_coords(data,combine_attrs='drop_conflicts')

The NetCDF files are attached:

datafiles.zip

[jrmagers]

One workaround (as suggested in #5194) is to use combine_nested():

import xarray as xr
import pathlib

files = pathlib.Path('.').glob('*.nc')

data = list()
for f in files:
    data.append(xr.open_dataset(f))

# combine_nested() works
nested_list = [[data[0],data[1]],
               [data[2],data[3]]]

combined_data = xr.combine_nested(nested_list,
                                  concat_dim=['tune_freq','ref_level'],
                                  combine_attrs='drop_conflicts')

combined_data = combined_data.sortby(['ref_level','tune_freq'])
combined_data.evm.plot()

sortby() is required in order to be able to plot and to avoid:

ValueError: The input coordinate is not sorted in increasing order along axis 0. This can lead to unexpected results. Consider calling the `sortby` method on the input DataArray. To plot data with categorical axes, consider using the `heatmap` function from the `seaborn` statistical plotting library.

But I still can't understand why this doesn't work with combine_by_coords().

[max-sixty]

Could I clarify the question — is this a feature request for combining non-monotonic indexes with combine_by_coords? Or are they actually monotonic?

[jrmagers]

@max-sixty: This is not a feature request, more of a help request :)

I believe the indexes are monotonic. I attempted to verify this by differentiation:

np.diff(data[1].ref_level.values)

[max-sixty]

Could you show the results of that? Are they globally monotonic?

[jrmagers]

Each dataset's dimensional coords are monotonic:

import xarray as xr
import pathlib
import numpy as np

files = pathlib.Path('.').glob('*.nc')

data = list()
for f in files:
    d = xr.open_dataset(f)
    print(np.diff(d.ref_level.values))
    print(np.diff(d.tune_freq.values))

Output:

[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
 1.]
[0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
 1. 1. 1. 1. 1. 1. 1. 1.]
[0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[0.5 0.5 0.5 0.5 0.5 0.5]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[0.5 0.5 0.5 0.5 0.5 0.5]

Globally, I don't think they are monotonic. The error returned by _combine_single_variable_hypercube() is correct. The 'ref_level' coords look like this right before the error is thrown:

[-50. -49. -48. -47. -46. -45. -44. -43. -42. -41. -40. -39. -38. -37.
 -36. -35. -34. -33. -32. -31. -30. -29. -28. -27. -26. -25. -24. -23.
 -22. -21. -20. -19. -18. -17. -16. -15. -14. -13. -12. -11. -10.  -9.
  -8.  -7.  -6.  -5.  -4.  -3.  -2.  -1.  -2.  -1.   0.   1.   2.   3.
   4.   5.   6.   7.   8.   9.  10.  11.  12.  13.  14.  15.  16.  17.
  18.  19.  20.  21.  22.  23.  24.  25.  26.  27.  28.  29.  30.  31.
  32.  33.  34.  35.  36.   0.   1.   2.   3.   4.   5.   6.   7.   8.
   9.  10.  11.  12.  13.  14.  15.  16.  17.  18.  19.  20.  21.  22.
  23.  24.  25.  26.  27.  28.  29.  30.  31.  32.]

You can see that the combined ("global") ref_level coordinates go from -50, -49, ... ,35, 36, 0, 1, ... , 31, 32. So it would appear that the coordinates are not being combined correctly.

[TomNicholas]

Hi @jrmagers - thanks for this clear and reproducible bug report.

Whilst the coordinates are monotonic (the easiest way to check is to extract the underlying pandas index via ds.ref_level.to_index().is_monotonic), I think the problem is caused by the fact that your datasets do not have consistent lengths along ref_level. In fact

file 0B.nc has length 33 along 'ref_level'
file 1A.nc has length 48 along 'ref_level'
file 1B.nc has length 39 along 'ref_level'
file 0A.nc has length 50 along 'ref_level'

The combining is failing when it tries to decide how to align each tile, because the tiles are of different sizes, so they can't simply be lined up next to one another like you probably expected. How it handles alignment is dictated through the options that get sent down to xr.concat() and eventually xr.align(), in particular the join kwarg. By default join=outer, which I think is what's creating the non-monotonic result you're seeing, as it's attempting to fill in the "gaps" between your un-alignable tiles.

If you try again but passing join='exact' then you get a clearer error:

Traceback (most recent call last):
  File "combine.py", line 16, in <module>
    combined_data = xr.combine_by_coords(data, join='exact', combine_attrs='drop_conflicts')
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/combine.py", line 897, in combine_by_coords
    concatenated = _combine_single_variable_hypercube(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/combine.py", line 614, in _combine_single_variable_hypercube
    concatenated = _combine_nd(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/combine.py", line 218, in _combine_nd
    combined_ids = _combine_all_along_first_dim(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/combine.py", line 254, in _combine_all_along_first_dim
    new_combined_ids[new_id] = _combine_1d(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/combine.py", line 277, in _combine_1d
    combined = concat(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/concat.py", line 242, in concat
    return f(
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/concat.py", line 440, in _dataset_concat
    align(*datasets, join=join, copy=False, exclude=[dim], fill_value=fill_value)
  File "/home/tom/Documents/Work/Code/xarray/xarray/core/alignment.py", line 323, in align
    raise ValueError(f"indexes along dimension {dim!r} are not equal")
ValueError: indexes along dimension 'ref_level' are not equal

I'm not quite sure how you wanted combine_by_coords to handle this situation, but one solution would be to trim (or pad) your data such that each dataset had the same length along the ref_level dimension. (Note it's only ref_level that's a problem, tune_freq can be aligned fine because

file 0B.nc has length 17 along 'tune_freq'
file 1A.nc has length 7 along 'tune_freq'
file 1B.nc has length 7 along 'tune_freq'
file 0A.nc has length 17 along 'tune_freq'

This was hard to spot from the error raised though - we could probably do with having either clearer error reporting, or more strict defaults. I wonder if we should make join='exact' the default for combine_by_coords... Another possibility is that actually the result of join='outer' can sometimes be non-montonic globally, and that we shouldn't demand monotonicity of the result when using join='outer'. I haven't really thought hard enough to know if that's okay though.

[jrmagers]

@TomNicholas,

Thanks for the explanation. Ultimately, my "tiles" look like this:

Which explains why combined_nested works well since it is first stacking/concatenating the tiles first along the ref_level coordinate and then along the tune_freq coordinate. The tune_freq coordinate is constant vs ref_level in all cases. In this case, it wouldn't make sense to trim or pad each dataset such that each one had the same dimension along ref_level.

Thus, given this arrangement of the tiles, combine_nested looks like the right choice to me.

[TomNicholas]

in all cases. In this case, it wouldn't make sense to trim or pad each dataset such that each one had the same dimension along ref_level.

I see, that diagram makes it much clearer!

Thus, given this arrangement of the tiles, combine_nested looks like the right choice to me.

I agree.