sel broadcasting(?) with missing values

[ahuang11]

I am trying to align observed valid times which are 1-dimensional (T) with forecast valid times which are 2-dimensional, init (S)+ lead (L).

# obs
<xarray.Dataset>
Dimensions:  (T: 468, X: 360, Y: 180)
Coordinates:
  * Y        (Y) float32 -89.5 -88.5 -87.5 -86.5 -85.5 ... 86.5 87.5 88.5 89.5
  * X        (X) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
  * T        (T) float32 262.5 263.5 264.5 265.5 ... 726.5 727.5 728.5 729.5
Data variables:
    sst      (T, Y, X) float32 ...

# cfs
<xarray.Dataset>
Dimensions:  (L: 10, S: 348)
Coordinates:
  * L        (L) float32 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
  * S        (S) float32 264.0 265.0 266.0 267.0 ... 608.0 609.0 610.0 611.0
Data variables:
    T        (L, S) float32 264.5 265.5 266.5 267.5 ... 617.5 618.5 619.5 620.5

Using .sel yields the result I want,

import numpy as np
import pandas as pd
import xarray as xr

cfs_url = 'http://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/.NCEP-CFSv2/.HINDCAST/.MONTHLY/.T/dods'
obs_url = "http://iridl.ldeo.columbia.edu/expert/SOURCES/.NOAA/.NCEP/.EMC/.CMB/.GLOBAL/.Reyn_SmithOIv2/.monthly/.sst/T/1+index/dods"

obs = xr.open_dataset(obs_url, decode_times=False)
cfs = xr.open_dataset(cfs_url, decode_times=False)

obs.sel(T=cfs['T'])
<xarray.Dataset>
Dimensions:  (L: 10, S: 348, X: 360, Y: 180)
Coordinates:
  * Y        (Y) float32 -89.5 -88.5 -87.5 -86.5 -85.5 ... 86.5 87.5 88.5 89.5
  * X        (X) float32 0.5 1.5 2.5 3.5 4.5 ... 355.5 356.5 357.5 358.5 359.5
    T        (L, S) float64 264.5 265.5 266.5 267.5 ... 617.5 618.5 619.5 620.5
  * L        (L) float32 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
  * S        (S) float32 264.0 265.0 266.0 267.0 ... 608.0 609.0 610.0 611.0
Data variables:
    sst      (L, S, Y, X) float32 ...
Attributes:
    Conventions:  IRIDL

However, it breaks KeyError: "not all values found in index 'T'" if obs doesn't have all the valid times of the forecast, so I figured I can simply interpolate.

obs = xr.open_dataset(obs_url, decode_times=False).isel(T=slice(0, 10))
cfs = xr.open_dataset(cfs_url, decode_times=False)
obs.interp(T=cfs['T'])

But if I want the fill values to be NaN, I have to re-mask it. Is there a more straightforward way without using where, e.g. sel(fill_value=np.nan)?

obs = xr.open_dataset(obs_url, decode_times=False).isel(T=slice(0, 10))
cfs = xr.open_dataset(cfs_url, decode_times=False)
tst = obs.interp(T=cfs['T'])
tst.where(tst['T'].isin(obs['T'].values))

I tried using align, but it seems to have no effect (just outputting the input without change).

[keewis]

I would use reindex for that:

obs.reindex(T=cfs['T'])

[ahuang11]

Thanks for the suggestion. Unfortunately, reindex doesn't work for me since CFS T is 2 dimension while OBS is 1 dimension.

[dcherian]

I think this is a good feature request for .sel(method="fill_value", fill_value=np.nan) or even when method=None and fill_value is not None

[max-sixty]

What do you see as the difference between reindex and sel? This goes in the direction of combining them — which might be fine.

A competing proposal would allow reindex to take a multidimensional object like sel does

[ahuang11]

I think both combining them with reindex automatically defaulting to fill_value=np.nan (similar to .map and .apply) and allowing reindex to take a multidimensional object is a good idea since sel is used much more prevalently in xarray than reindex (I think), but if users came from pandas, they may be familiar with reindex.

[dcherian]

@max-sixty Yes I guess there is substantial overlap between sel and reindex. Your proposal also sounds good to me.

[mathause]

Does

obs_cut = obs.isel(T=slice(0, 10))
obs_cut.sel(T=cfs['T'], method="nearest")

do what you need?

[ahuang11]

Thanks for the suggestion. It does not because if the forecast valid/target time was 2020-02-01 and the latest observation is 2020-01-01, using a month earlier observation to verify is not too useful. I think the functionality I want is reindex, but it doesn't support multidimensional objects.