Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set order='F' when raveling group_idx after broadcast #286

Merged
merged 5 commits into from
Nov 8, 2023
Merged

Conversation

dcherian
Copy link
Collaborator

@dcherian dcherian commented Nov 8, 2023

This majorly improves the dim=... case for engine="flox" at least. xref #281

I'm not sure if it is a regression for engine="numpy": EDIT it is a small regression, but a big one for engine="numbagg" (9ms -> 27ms).

We trade off a single bad reshape for array against argsorting both array and group_idx for a ~10-20x speedup

ds = xr.tutorial.load_dataset('air_temperature')
ds.groupby('lon').count(..., engine="flox")

This majorly improves the dim=... case for engine="flox" at least.
xref #281

I'm not sure if it is a regression for engine="numpy"

We trade off a single bad reshape for array against argsorting both
array and group_idx for a ~10-20x speedup

```
ds = xr.tutorial.load_dataset('air_temperature')
ds.groupby('lon').count(..., engine="flox")
```
@dcherian
Copy link
Collaborator Author

dcherian commented Nov 8, 2023

| Before [0cea9721] | After [375a9f7e] | Ratio | Benchmark (Parameter)                                              |
| <main>            | <opt-ravel>      |       |                                                                    |
|-------------------+------------------+-------+--------------------------------------------------------------------|
| 189±8ms           | 90.0±0.9ms       |  0.48 | reduce.ChunkReduce2DAllAxes.time_reduce('nanmean', 'bins', 'flox') |
| 160±10ms          | 74.7±0.3ms       |  0.47 | reduce.ChunkReduce2DAllAxes.time_reduce('nanmax', 'bins', 'flox')  |
| 160±8ms           | 75.3±0.6ms       |  0.47 | reduce.ChunkReduce2DAllAxes.time_reduce('nansum', 'bins', 'flox')  |
| 174±2ms           | 76.7±3ms         |  0.44 | reduce.ChunkReduce2DAllAxes.time_reduce('count', 'bins', 'flox')   |
| 178±9ms           | 79.2±1ms         |  0.44 | reduce.ChunkReduce2DAllAxes.time_reduce('mean', 'bins', 'flox')    |
| 158±6ms           | 65.9±0.7ms       |  0.42 | reduce.ChunkReduce2DAllAxes.time_reduce('sum', 'bins', 'flox')     |
| 165±9ms           | 66.1±2ms         |   0.4 | reduce.ChunkReduce2DAllAxes.time_reduce('max', 'bins', 'flox')     |
| 506±2ms           | 51.0±1ms         |   0.1 | reduce.ChunkReduce2DAllAxes.time_reduce('nanmean', 'None', 'flox') |
| 510±2ms           | 53.4±1ms         |   0.1 | reduce.ChunkReduce2DAllAxes.time_reduce('mean', 'None', 'flox')    |
| 539±20ms          | 40.5±0.2ms       |  0.08 | reduce.ChunkReduce2DAllAxes.time_reduce('nanmax', 'None', 'flox')  |
| 495±4ms           | 38.1±1ms         |  0.08 | reduce.ChunkReduce2DAllAxes.time_reduce('nansum', 'None', 'flox')  |
| 497±2ms           | 35.3±0.5ms       |  0.07 | reduce.ChunkReduce2DAllAxes.time_reduce('count', 'None', 'flox')   |
| 495±3ms           | 34.9±0.2ms       |  0.07 | reduce.ChunkReduce2DAllAxes.time_reduce('max', 'None', 'flox')     |
| 542±20ms          | 26.8±0.9ms       |  0.05 | reduce.ChunkReduce2DAllAxes.time_reduce('sum', 'None', 'flox')     |

@dcherian dcherian enabled auto-merge (squash) November 8, 2023 17:50
@dcherian dcherian merged commit 273d319 into main Nov 8, 2023
19 checks passed
@dcherian dcherian deleted the opt-ravel branch November 8, 2023 17:54
dcherian added a commit that referenced this pull request Nov 8, 2023
* main:
  Set order='F' when raveling group_idx after broadcast (#286)
  Ignore benchmarks for codecov (#287)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant