Improve concat performance #7824

Illviljan · 2023-05-07T14:54:06Z

Don't use python for loops for possibly large coords. Rather create a np array once then filter out bad data.
DuckArrayModule is slightly slow, so cache the first import in a dict instead to speed up later calls.
Add more typing to be more confident that inputs are valid and then remove redundant checks and conversions.

Requires Improve interp performance #7843
Requires Improve to_dask_dataframe performance #7844.
Requires Add type hints to test_dtypes #7858
Closes Slow performance of concat() #7833
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst

for more information, see https://pre-commit.ci

…into improve_concat

for more information, see https://pre-commit.ci

…into improve_concat

xarray/core/variable.py

xarray/core/merge.py

asv_bench/benchmarks/combine.py

xarray/core/concat.py

doc/whats-new.rst

xarray/core/concat.py

xarray/core/dataset.py

xarray/core/indexes.py

dcherian

LGTM. Thanks @Illviljan Great PR!

* 1. var_idx very slow * 2. slow any * Add test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 3. Slow array_type called multiple times * 4. Can use fastpath for variable.concat? * 5. slow init of pd.unique * typos * Update concat.py * Update merge.py * 6. Avoid recalculating in loops * 7. No need to transpose 1d arrays. * 8. speed up dask_dataframe * Update dataset.py * Update dataset.py * Update dataset.py * Add dask combine test with many variables * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update combine.py * Update combine.py * Update combine.py * list not needed * dim is usually string, might be faster to check for that * first_var.dims doesn't change and can be defined 1 time * mask bad points rather than append good points * reduce duplicated code * don't think id() is required here. * get dtype directly instead of through result_dtype * seems better to delete rather than append, * use internal fastpath if it's a dataset, values should be fine then * Change isinstance order. * use fastpath if already xarray objtect * Update variable.py * Update dtypes.py * typing fixes * more typing fixes * test undoing as_compatible_data * undo concat_dim_length deletion * Update xarray/core/concat.py * Remove .copy and sum * Update concat.py * Use OrderedSet * Apply suggestions from code review * Update whats-new.rst * Update xarray/core/concat.py * no need to check arrays if cupy isnt even installed * Update whats-new.rst * Add concat comment * minimize diff * revert sketchy --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

1. var_idx very slow

670d970

Illviljan added the run-benchmark Run the ASV benchmark workflow label May 7, 2023

Illviljan added 2 commits May 7, 2023 17:21

2. slow any

1370a0e

Add test

8ab83fa

github-actions bot added topic-arrays related to flexible array support topic-performance labels May 7, 2023

pre-commit-ci bot and others added 14 commits May 7, 2023 16:04

[pre-commit.ci] auto fixes from pre-commit.com hooks

b6e1881

for more information, see https://pre-commit.ci

3. Slow array_type called multiple times

487e958

4. Can use fastpath for variable.concat?

fbb5430

Merge branch 'improve_concat' of https://github.com/Illviljan/xarray …

e6fe2c4

…into improve_concat

5. slow init of pd.unique

dc5f0e6

typos

38aa169

Update concat.py

5fd6bcb

Update merge.py

43dcff2

6. Avoid recalculating in loops

7ef0e5d

7. No need to transpose 1d arrays.

b2c067d

Merge branch 'main' into improve_concat

914481b

8. speed up dask_dataframe

ad048b6

Update dataset.py

d609883

Update dataset.py

4005f6f

Illviljan mentioned this pull request May 11, 2023

Slow performance of concat() #7833

Closed

Illviljan and others added 9 commits May 11, 2023 05:39

Update dataset.py

d23c833

Add dask combine test with many variables

6c6b5c7

[pre-commit.ci] auto fixes from pre-commit.com hooks

068ba55

for more information, see https://pre-commit.ci

Merge branch 'main' into improve_concat

1bb9dd5

Merge branch 'improve_concat' of https://github.com/Illviljan/xarray …

6828fb6

…into improve_concat

Update combine.py

5670331

Update combine.py

b11afe8

Update combine.py

e1938a8

list not needed

43fd7a2

Illviljan added 2 commits May 16, 2023 23:07

Remove .copy and sum

86eb72a

Merge branch 'main' into improve_concat

b2a498a

dcherian reviewed May 18, 2023

View reviewed changes

xarray/core/variable.py Show resolved Hide resolved

Illviljan added 2 commits May 18, 2023 23:58

Update concat.py

ee2c3f6

Merge branch 'main' into improve_concat

4092d1b

Illviljan mentioned this pull request May 21, 2023

Avoid explicit loop when updating OrderedSet #7857

Merged

Illviljan added 3 commits May 21, 2023 11:17

Use OrderedSet

1e514f6

Merge branch 'main' into improve_concat

59bf15b

Merge branch 'main' into improve_concat

b4198a3

Illviljan commented May 25, 2023

View reviewed changes

Illviljan added 3 commits May 25, 2023 22:48

Apply suggestions from code review

0d0b76e

Merge branch 'main' into improve_concat

51768bd

Update whats-new.rst

15e2783

Illviljan commented May 31, 2023

View reviewed changes

xarray/core/concat.py Outdated Show resolved Hide resolved

Illviljan added 3 commits May 31, 2023 17:58

Update xarray/core/concat.py

6f86e98

no need to check arrays if cupy isnt even installed

db25a0b

Merge branch 'main' into improve_concat

da959a8

dcherian reviewed Jun 1, 2023

View reviewed changes

doc/whats-new.rst Outdated Show resolved Hide resolved

xarray/core/concat.py Show resolved Hide resolved

xarray/core/dataset.py Show resolved Hide resolved

xarray/core/indexes.py Outdated Show resolved Hide resolved

Illviljan added 4 commits June 1, 2023 19:16

Update whats-new.rst

629cdea

Add concat comment

cfaa866

minimize diff

fe62336

revert sketchy

4e90504

dcherian approved these changes Jun 1, 2023

View reviewed changes

Illviljan added the plan to merge Final call for comments label Jun 2, 2023

dcherian merged commit c9d89e2 into pydata:main Jun 2, 2023

dcherian mentioned this pull request Jun 15, 2023

Comprehensive benchmarking suite #4648

Open

19 tasks

This was referenced Jul 7, 2023

Merging 'coordinates' can be slow, could be skipped? (performance) boutproject/xBOUT#220

Open

Plotting responsiveness - optimizing from_region()? boutproject/xBOUT#234

Open

xarray - compatibility with latest versions boutproject/xBOUT#275

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve concat performance #7824

Improve concat performance #7824

Illviljan commented May 7, 2023 •

edited

Loading

dcherian left a comment

Improve concat performance #7824

Improve concat performance #7824

Conversation

Illviljan commented May 7, 2023 • edited Loading

dcherian left a comment

Choose a reason for hiding this comment

Illviljan commented May 7, 2023 •

edited

Loading