Add temporal utilities and averaging functionalities

[tomvothecoder]

Description

• Closes Add temporal averaging functionality (time series avg, climatology and departure) #47

This PR adds temporal averaging functionality, including calculating time series averaging, climatology, and departures.

Summary of Changes

• Operates on a data variable within a dataset, and returns a dataset
• Stores metadata regarding temporal averaging in the data variable's attributes
• Supported grouping frequencies:
  • Time series average:
    • "year", "season", "month", "day", "hour"
      • "year" groups by year
      • "season" groups by (year, season)
      • "month" groups by (year, month)
      • "day" groups by (year, month, day)
      • "hour" groups by (year, month, day, hour)
  • Climatology and departures:
    • "season", "month", "day"
      • "season" groups by season
      • "month" groups by month
      • "day" groups by (month, day)
• Support for weighted and unweighted averaging (defaults to True)
• Support for centering time coordinates using time bounds (defaults to true)
• Configuration specific to "season" frequency
  • Predefined seasons:
    • "dec_mode" - mode for the seasons that includes December ("DJF" or "JFD") (defaults to "DJF")
    • "drop_incomplete_djf" - if "dec_mode" is "DJF", drop incomplete "DJF" seasons (defaults to False)
  • Custom seasons:
    • "custom_seasons" - list of lists containing the 3-letter string representation of months (e.g., "Jan")

Validation

If you are interested in the validation of this feature against CDAT, please checkout the notebooks found here: xCDAT/xcdat-validation#28

Future Enhancements

Feature Specification Documentation: https://docs.google.com/document/d/1klHh5LLYcmNSopvptSVYmcgtUVLFnLMueRhfCF6tbaU/edit#

• Time series averaging frequency - "nHour"
• Climatology frequency - "diurnalNN"
• [FEATURE]: Add averages with time dimension removed #201
• [FEATURE]: Replace MultiIndex outputs for time series avg/climatology time coordinates with datetime #202
• [Feature]: Support Climatology bounds #85
• Update seasonal cycle departures to group on year and season like CDAT for seasonal anomalies
• Handling missing data
  • User may want to set criteria for excluding locations entirely from a climatology if the amount of missing data is excessive or unrepresentative.
  • CDAT allowed the user to set two criteria for excluding a location (i.e., setting it to “missing”) to mask out locations with insufficient data.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

If applicable:

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[codecov-commenter]

Codecov Report

Merging #107 (ff1341f) into main (201a575) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              main      #107    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            8         9     +1     
  Lines          381       741   +360     
==========================================
+ Hits           381       741   +360     

Impacted Files	Coverage Δ
xcdat/__init__.py	100.00% <100.00%> (ø)
xcdat/axis.py	100.00% <100.00%> (ø)
xcdat/bounds.py	100.00% <100.00%> (ø)
xcdat/dataset.py	100.00% <100.00%> (ø)
xcdat/spatial_avg.py	100.00% <100.00%> (ø)
xcdat/temporal.py	100.00% <100.00%> (ø)
xcdat/utils.py	100.00% <100.00%> (ø)
xcdat/xcdat.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93a923e...ff1341f. Read the comment docs.

[chengzhuzhang]

Hey @chengzhuzhang, ds.xcdat.center_time() returns a dataset, so you need to update the variable ds.

Thank you, Tom. center_time is very useful. I'm wondering is it straightforward to make it as an argument in open_dataset().

[tomvothecoder]

Hey @chengzhuzhang, ds.xcdat.center_time() returns a dataset, so you need to update the variable ds.

Thank you, Tom. center_time is very useful. I'm wondering is it straightforward to make it as an argument in open_dataset().

No problem, glad to hear! Yes, adding center_time as a kwarg to open_dataset() is straightforward so I'll do that.

[tomvothecoder]

Some minor review updates

[durack1]

@tomvothecoder this looks great, I wonder are all the various flavours of calendar managed by XCDAT?

[tomvothecoder]

Thanks @durack1! Xarray uses cftime under the hood, so XCDAT should be capable of handling the various calendar types if we implement features using primarily the xarray APIs.

https://xarray.pydata.org/en/stable/user-guide/weather-climate.html#non-standard-calendars-and-dates-outside-the-timestamp-valid-range

https://xarray.pydata.org/en/stable/user-guide/time-series.html#creating-datetime64-data

When decoding/encoding datetimes for non-standard calendars or for dates before year 1678 or after year 2262, xarray uses the cftime library. It was previously packaged with the netcdf4-python package under the name netcdftime but is now distributed separately. cftime is an optional dependency of xarray.

[durack1]

Ok this is great, it seems all the cdtime calendars are included in cftime, so we're good, great!

So the cftime library includes (https://unidata.github.io/cftime/api.html#cftime.num2date)

calendar: describes the calendar used in the time calculations. All the values currently
defined in the [CF metadata convention](http://cfconventions.org/cf-conventions/cf-conventions#calendar)
are supported. Valid calendars ‘standard’, ‘gregorian’, ‘proleptic_gregorian’ ‘noleap’,
‘365_day’, ‘360_day’, ‘julian’, ‘all_leap’, ‘366_day’. Default is ‘standard’, which is a
mixed Julian/Gregorian calendar.

Just as an FYI, dropping some cdtime links here for documentation:
https://github.com/CDAT/cdtime/blob/master/Src/cdtimemodule.c#L1422-L1430 <- cdtime calendar types,

    PyDict_SetItemString(d, "error", PyCdtime_ErrorObject); /* export exception */
    DefineLongConstant(d, "StandardCalendar", (long)cdStandard);
    DefineLongConstant(d, "GregorianCalendar", (long)cdStandard);
    DefineLongConstant(d, "JulianCalendar", (long)cdJulian);
    DefineLongConstant(d, "MixedCalendar", (long)cdMixed);
    DefineLongConstant(d, "NoLeapCalendar", (long)cdNoLeap);
    DefineLongConstant(d, "Calendar360", (long)cd360);
    DefineLongConstant(d, "ClimCalendar", (long)cdClim);
    DefineLongConstant(d, "ClimLeapCalendar", (long)cdClimLeap);
    DefineLongConstant(d, "DefaultCalendar", (long)cdMixed); /* initialize the default calendar */

Which calls
https://github.com/CDAT/libcdms/blob/master/include/cdms.h#L149-L167

#define cdStandardCal   0x11
#define cdClimCal        0x0
#define cdHasLeap      0x100
#define cdHasNoLeap    0x000
#define cd365Days     0x1000
#define cd360Days     0x0000
#define cdJulianCal  0x10000
#define cdMixedCal   0x20000

typedef enum cdCalenType {
	cdStandard    = ( cdStandardCal | cdHasLeap   | cd365Days),
	cdJulian      = ( cdStandardCal | cdHasLeap   | cd365Days | cdJulianCal),
	cdNoLeap      = ( cdStandardCal | cdHasNoLeap | cd365Days),
	cd360         = ( cdStandardCal | cdHasNoLeap | cd360Days),
	cdClim        = ( cdClimCal     | cdHasNoLeap | cd365Days),
	cdClimLeap    = ( cdClimCal     | cdHasLeap   | cd365Days),
	cdClim360     = ( cdClimCal     | cdHasNoLeap | cd360Days),
	cdMixed       = ( cdStandardCal | cdHasLeap   | cd365Days | cdMixedCal)
}  cdCalenType;

[tomvothecoder]

Hey everyone, this PR is ready for a final round of review. Here are the associated validation notebooks that I mentioned on our email chain: xCDAT/xcdat-validation#28

@lee1043, @chengzhuzhang, and @pochedls -- feel free to try out the API and look through the notebooks
@jasonb5 -- it will be nice if you can review the code for any optimizations, improvements, and fixes

I'm hoping to have this PR ready to be merged by our meeting next Wednesday. Thanks!

[chengzhuzhang]

Hi @tomvothecoder ,

My testing goes smoothly so far. Only have two high level comments:

• It would be really handy, if this feature can include the capability to calculate weighted temporal average. Right now in xarray, one still has to generates the weights and then do something like (d * weights).groupby("time.month").sum(dim="time") I think temporal_average can cover this case straightforwardly? perhaps we can add a mean mode in addition or enable this if neither "climatology" nor "timeseries" is supplied?
• And have a time coordinate, as well as a multi-index coordinate is a bit confusing. The additional step to convert the multi-index coordinates back to a new time coordinate seems a good way to clear this confusing. But it will require some additional work...

[durack1]

Folks I just came across this, might be worth a quick look to see alternatives https://ncar.github.io/esds/posts/2021/yearly-averages-xarray/

@chengzhuzhang @tomvothecoder @pochedls

[jasonb5]

Rest of the code LGTM.

The requested changes have been made and doesn't require a re-review.

[tomvothecoder]

Hi @tomvothecoder ,

My testing goes smoothly so far. Only have two high level comments:

* It would be really handy, if this feature can include the capability to calculate weighted temporal average. Right now in `xarray`, one still has to generates the `weights` and then do something like `(d * weights).groupby("time.month").sum(dim="time")` I think `temporal_average` can cover this case straightforwardly? perhaps we can add a `mean` mode in addition or enable this if neither "climatology" nor "timeseries" is supplied?

* And have a time coordinate, as well as a multi-index coordinate is a bit confusing. The additional step to convert the multi-index coordinates back to a new time coordinate seems a good way to clear this confusing. But it will require some additional work...

I decided to carry these issues over to #201 and #202.

[chengzhuzhang]

Sounds good!