Comprehensive benchmarking suite

[dcherian]

I think a good "infrastructure" target for the NASA OSS call would be to expand our benchmarking suite (https://pandas.pydata.org/speed/xarray/#/)

AFAIK running these in a useful manner on CI is still unsolved (please correct me if I'm wrong). But we can always run it on an NCAR machine using a cron job.

Thoughts?

cc @scottyhq

A quick survey of work needed (please append):

• indexing & slicing Improve indexing performance benchmarks #3382 Performance: numpy indexes small amounts of data 1000 faster than xarray #2799 Slow performance of isel #2227
• DataArray construction Speed up Dataset._construct_dataarray #4744
• attribute access Attribute style access is slow #4741, speedup attribute style access and tab completion #4742
• property access Should we cache some small properties? #3514
• reindexing? slow performance with open_mfdataset #1385 (comment)
• alignment Performance problem when doing computation between two arrays with discontinuous indexes  #3755, [skip-ci] Add alignment benchmarks #7738
• assignment Needs performance check / improvements in value assignment of DataArray #1771
• coarsen
• groupby groupby very slow compared to pandas #659 [skip-ci] Add cftime groupby, resample benchmarks #7795 Speed up .dt accessor by preserving Index objects. #7796
• resample Resample is ~100x slower than Pandas resample; Speed is related to resample period (unlike Pandas) #4498 [skip-ci] Add cftime groupby, resample benchmarks #7795
• weighted Allow skipna in .dot() #4482 weighted operations: performance optimisations #3883
• concat Improve concat performance #7824
• merge
• open_dataset, open_mfdataset We need a fast path for open_mfdataset #1823
• stack / unstack
• apply_ufunc?
• interp Faster interp #4740 Improve interp performance #7843
• reprs Speed up Dataset._construct_dataarray #4744
• to_(dask)_dataframe Improve to_dask_dataframe performance #7844 Add benchmarks for to_dataframe and to_dask_dataframe #7474

Related: #3514

[scottyhq]

thanks for the ping @dcherian, i really like the idea! One other thing that often gets neglected in test suites is operating on remote data. I understand the need to avoid long-running tests and tests prone to network failures for PRs, but running these sorts of examples as a cron job could be very helpful for benchmarking and detecting issues.

In intake-xarray we recently added tests against a local HTTP server and "S3" server:
https://github.com/intake/intake-xarray/blob/master/intake_xarray/tests/test_remote.py

Also added several simple tests requiring a network connection to public data (no auth required) that we run locally but not in CI currently:
https://github.com/intake/intake-xarray/blob/master/intake_xarray/tests/test_network.py

[dcherian]

Thanks @scottyhq

One other thing that often gets neglected in test suites is operating on remote data.

This is lining up with the "pangeo integration tests" that came up in a Pangeo meeting (cc @rabernat).

Regardless whether it fits, I think adding benchmarks+tests for the xarray+zarr+fsspec (or xarray+mfdataset+netCDF) is an important and unmet need of the Pangeo community in general that we could address.

[max-sixty]

This would be great.

Down a couple of levels — I think potentially we could run this as a cron job on GitHub Actions. NCAR would also be a good plan. I'm also happy to supply a VM if that's helpful.

[dcherian]

Looks like Quansight thinks that GH actions is a good place to benchmark scikit-learn: https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/ so may be we can set that up for our existing benchmarks.

Here's the workflow: https://github.com/jaimergp/scikit-image/blob/main/.github/workflows/benchmarks-cron.yml

[dcherian]

@TomAugspurger are you still in charge of the pydata benchmarking machine? If so, could you add xarray to the list please (https://pandas.pydata.org/speed/)? @Illviljan has made major improvements so it should be a lot faster now

[TomAugspurger]

"In charge of" is overstating it a bit. It's been segfaulting when building pandas and I haven't had a chance to debug it.

If / when I get around to fixing it I'll try adding xarray, but it might be a bit.