Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: Improve performance in rolling.mean(engine=numba) #44176

Merged
merged 6 commits into from
Oct 28, 2021

Conversation

mroeschke
Copy link
Member

  • tests added / passed
  • Ensure all linting tests pass, see here for how to run them
  • whatsnew entry

This also starts to add a shared aggregation function (sum) that can shared between rolling/groupby/DataFrame when using the numba engine.

df = pd.DataFrame(np.ones((10000, 1000)))
roll = df.rolling(10)
roll.sum(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})
%timeit roll.sum(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

211 ms ± 12.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- PR
424 ms ± 9.23 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- master

@mroeschke mroeschke added Performance Memory or execution speed performance Window rolling, ewma, expanding labels Oct 25, 2021
@mroeschke mroeschke added this to the 1.4 milestone Oct 25, 2021
@jreback jreback merged commit 5d520c6 into pandas-dev:master Oct 28, 2021
@jreback
Copy link
Contributor

jreback commented Oct 28, 2021

great, can you make sure that we have sufficient benchmarks

e.g.

  • cython vs numba parallel=True & False
  • small number of cols vs large number

@mroeschke mroeschke deleted the enh/rolling_numba_sum branch October 29, 2021 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Memory or execution speed performance Window rolling, ewma, expanding
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants