Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test for pl.len() on LazyFrames #21349

Closed
2 tasks done
CangyuanLi opened this issue Feb 19, 2025 · 2 comments · Fixed by #21418
Closed
2 tasks done

Add test for pl.len() on LazyFrames #21349

CangyuanLi opened this issue Feb 19, 2025 · 2 comments · Fixed by #21418
Labels
bug Something isn't working good first issue Good for newcomers python Related to Python Polars test Related to the test suite

Comments

@CangyuanLi
Copy link

CangyuanLi commented Feb 19, 2025

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.DataFrame({
    "id1": [1, 1, 1, 2, 3],
    "id2": [100, 100, 200, 100, 300],
})

print("DataFrame")
print(df.height)
print(df.unique().height)
print(df.unique("id1").height)

print(df.unique().sort(["id1", "id2"]))

# DataFrame
# 5
# 4
# 4
# 3
# shape: (4, 2)
# ┌─────┬─────┐
# │ id1 ┆ id2 │
# │ --- ┆ --- │
# │ i64 ┆ i64 │
# ╞═════╪═════╡
# │ 1   ┆ 100 │
# │ 1   ┆ 200 │
# │ 2   ┆ 100 │
# │ 3   ┆ 300 │
# └─────┴─────┘

print("LazyFrame")
lf = df.lazy()
print(lf.select(pl.len()).collect().item())
print(lf.unique().select(pl.len()).collect().item())
print(lf.select("id1").unique().select(pl.len()).collect().item())
# print(lf.unique("id1").select(pl.len()).collect().item())

print(lf.unique().collect().sort(["id1", "id2"]))

# LazyFrame
# 5
# 3
# 3
# shape: (4, 2)
# ┌─────┬─────┐
# │ id1 ┆ id2 │
# │ --- ┆ --- │
# │ i64 ┆ i64 │
# ╞═════╪═════╡
# │ 1   ┆ 100 │
# │ 1   ┆ 200 │
# │ 2   ┆ 100 │
# │ 3   ┆ 300 │
# └─────┴─────┘

Log output

Issue description

I am not sure the extent of the issue, but I noticed that pl.len(), in combination with .unique(), reports the wrong results on LazyFrames when doing unique by more than one column. The correct results are shown on the DataFrame. Note that .unique() itself seems to be fine, as the resulting DataFrames are equivalent. Also,

print(lf.unique("id1").select(pl.len()).collect().item())

reports a ColumnNotFoundError, which seems related to #21343

Expected behavior

The expected behavior is that the DataFrame heights match up with the LazyFrame pl.len().

Installed versions

--------Version info---------
Polars:              1.22.0
Index type:          UInt32
Platform:            Linux-5.14.0-284.97.1.el9_2.x86_64-x86_64-with-glibc2.28
Python:              3.10.13 (main, Sep 11 2023, 13:21:10) [GCC 11.2.0]
LTS CPU:             False

----Optional dependencies----
Azure CLI            <not installed>
adbc_driver_manager  <not installed>
altair               5.5.0
azure.identity       <not installed>
boto3                1.24.28
cloudpickle          3.1.1
connectorx           <not installed>
deltalake            <not installed>
fastexcel            0.11.6
fsspec               2024.12.0
gevent               <not installed>
google.auth          2.38.0
great_tables         0.16.1
matplotlib           3.8.3
numpy                1.23.5
openpyxl             <not installed>
pandas               2.2.1
pyarrow              15.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           2.0.37
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>
@CangyuanLi CangyuanLi added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Feb 19, 2025
@cmdlineluser
Copy link
Contributor

I can reproduce the mismatch on 1.22.0

Testing on main, I get matching results - so it may already be fixed. (no recent commit stands out to me from a quick glance)

# DataFrame
# 5
# 4
# 3
# LazyFrame
# 5
# 4
# 3

The ColumnNotFoundError seems to have been fixed: #21154

>>> lf.unique("id1").select(pl.len()).collect().item()
3

@MarcoGorelli MarcoGorelli added the test Related to the test suite label Feb 20, 2025
@MarcoGorelli
Copy link
Collaborator

thanks for the report

so it may already be fixed. (no recent commit stands out to me from a quick glance)

in that case, it may be good to add this snippet as a test, to make sure it doesn't accidentally regress in the future

@MarcoGorelli MarcoGorelli added the good first issue Good for newcomers label Feb 20, 2025
@MarcoGorelli MarcoGorelli changed the title pl.len() broken on LazyFrames Add test for pl.len() on LazyFrames Feb 20, 2025
@MarcoGorelli MarcoGorelli removed the needs triage Awaiting prioritization by a maintainer label Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers python Related to Python Polars test Related to the test suite
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants