Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collect_schema produces incorrect results for an aggregation function followed by .dt.*. in a group_by context. #19990

Closed
2 tasks done
christian-hnz opened this issue Nov 25, 2024 · 2 comments · Fixed by #19993
Assignees
Labels
accepted Ready for implementation bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@christian-hnz
Copy link

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

from datetime import date
import polars as pl

df = pl.DataFrame(
    {
        "id": ["a", "a", "a", "b", "b"],
        "date": [date(2000, i, 1) for i in (1, 4, 7, 6, 12)],
    }
)
min_date = pl.col("date").min()

df.group_by("id").agg(min_date.dt.day()).schema  # Schema({'id': String, 'literal': Int8}) 
df.lazy().group_by("id").agg(min_date.dt.day()).collect_schema()  # Schema({'id': String, 'literal': List(Int8)})

Log output

keys/aggregates are not partitionable: running default HASH AGGREGATION

Issue description

Methods from the .dt namespace lead to an incorrect conclusion by collect_schema when used in a group_by context (it works correctly when replacing the above .group_by("id").agg( by .select() and when preceded by an aggregation function.

  • The below example shows min but it behaves the same way with max, mean, median, ...
  • The example yields then same results when replacing day by month or year.

Expected behavior

Calling collect_schema on the LazyFrame should also yield Schema({'id': String, 'literal': Int8}) instead of Schema({'id': String, 'literal': List(Int8)}).

Installed versions

--------Version info---------
Polars:              1.15.0
Index type:          UInt32
Platform:            Linux-6.8.0-109048-tuxedo-x86_64-with-glibc2.39
Python:              3.12.7 (main, Oct 20 2024, 10:55:11) [GCC 13.2.0]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
boto3                <not installed>
cloudpickle          3.1.0
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               2024.10.0
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           3.9.2
nest_asyncio         <not installed>
numpy                2.0.2
openpyxl             <not installed>
pandas               2.2.3
pyarrow              18.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>
@christian-hnz christian-hnz added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Nov 25, 2024
@lmocsi
Copy link

lmocsi commented Nov 26, 2024

This seems to have the same root cause as my issue: #19984

@christian-hnz
Copy link
Author

Wow, that was fast. Thanks for fixing!

@c-peters c-peters added the accepted Ready for implementation label Dec 2, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Dec 2, 2024
@c-peters c-peters moved this from Ready to Done in Backlog Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
Archived in project
4 participants