Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

groupby_dynamic min aggregation behaves unexpectedly at DataFrame tail #7548

Closed
2 tasks done
finncatling opened this issue Mar 14, 2023 · 2 comments · Fixed by #7556
Closed
2 tasks done

groupby_dynamic min aggregation behaves unexpectedly at DataFrame tail #7548

finncatling opened this issue Mar 14, 2023 · 2 comments · Fixed by #7556
Labels
bug Something isn't working python Related to Python Polars

Comments

@finncatling
Copy link

Polars version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of Polars.

Issue description

When I use groupyby_dynamic and try to calculate the per-group minimum in a column, the calculated minima are incorrect for windows which partially overlap the bottom of the DataFrame. For these windows, the minimum from the last non-overlapping window is returned.

NB. the maxima are calculated as expected.

Reproducible example

import numpy as np
import polars as pl

pl.DataFrame({
    "time": np.arange(4),
    "value": np.arange(4)
}).groupby_dynamic(
    "time",
    every="1i",
    period="3i"
).agg(
    pl.col("value"),
    pl.col("value").min().alias("min_value"),
    pl.col("value").max().alias("max_value"),
)

#┌──────┬───────────┬───────────┬───────────┐
#│ time ┆ value     ┆ min_value ┆ max_value │
#│ ---  ┆ ---       ┆ ---       ┆ ---       │
#│ i64  ┆ list[i64] ┆ i64       ┆ i64       │
#╞══════╪═══════════╪═══════════╪═══════════╡
#│ 0    ┆ [0, 1, 2] ┆ 0         ┆ 2         │
#│ 1    ┆ [1, 2, 3] ┆ 1         ┆ 3         │
#│ 2    ┆ [2, 3]    ┆ 1         ┆ 3         │
#│ 3    ┆ [3]       ┆ 1         ┆ 3         │
#└──────┴───────────┴───────────┴───────────┘

Expected behavior

┌──────┬───────────┬───────────┬───────────┐
│ time ┆ value     ┆ min_value ┆ max_value │
│ ---  ┆ ---       ┆ ---       ┆ ---       │
│ i64  ┆ list[i64] ┆ i64       ┆ i64       │
╞══════╪═══════════╪═══════════╪═══════════╡
│ 0    ┆ [0, 1, 2] ┆ 0         ┆ 2         │
│ 1    ┆ [1, 2, 3] ┆ 1         ┆ 3         │
│ 2    ┆ [2, 3]    ┆ 2         ┆ 3         │
│ 3    ┆ [3]       ┆ 3         ┆ 3         │
└──────┴───────────┴───────────┴───────────┘

Installed versions

---Version info---
Polars: 0.16.13
Index type: UInt32
Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Python: 3.10.2 (main, Jan 17 2022, 11:10:19) [GCC 9.3.0]
---Optional dependencies---
numpy: 1.24.2
pandas: 1.5.3
pyarrow: 11.0.0
connectorx: <not installed>
deltalake: <not installed>
fsspec: <not installed>
matplotlib: 3.7.1
xlsx2csv: <not installed>
xlsxwriter: <not installed>
@finncatling
Copy link
Author

Much appreciated :)

@ritchie46
Copy link
Member

Thanks for opening the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Related to Python Polars
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants