Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dt.truncate is 3-4x slower in polars compared to pandas #16531

Closed
2 tasks done
Chuck321123 opened this issue May 27, 2024 · 3 comments · Fixed by #16615
Closed
2 tasks done

dt.truncate is 3-4x slower in polars compared to pandas #16531

Chuck321123 opened this issue May 27, 2024 · 3 comments · Fixed by #16615
Labels
A-temporal Area: date/time functionality performance Performance issues or improvements python Related to Python Polars

Comments

@Chuck321123
Copy link

Chuck321123 commented May 27, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import pandas as pd
import polars as pl

num_rows = 1000000
utc_time = pd.date_range(start='2023-01-01', periods=num_rows, freq='min')

# Create the DataFrame
df = pd.DataFrame({
    'UTC_Time': utc_time
})

df['UTC_Time'] = df['UTC_Time'].sort_values()

# Display the first few rows
print(df.head())

Interval = 2

%timeit df["UTC_Time"].dt.floor(f'{Interval}h')

df = pl.DataFrame(df)

%timeit df.with_columns(pl.col("UTC_Time").dt.truncate(f"{Interval}h"))

Log output

8.7 ms ± 1.38 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
32.1 ms ± 915 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Issue description

So in my testings I get that truncate is way slower than the pandas equivalent. Do you guys also get the same results? In that case, it would be nice if someone could find a way to speed up this function.

Expected behavior

That dt.truncate becomes faster than the pandas equivalent

Installed versions

--------Version info---------
Polars:               0.20.30
Index type:           UInt32
Platform:             Windows-11-10.0.22631-SP0
Python:               3.12.2 | packaged by Anaconda, Inc. | (main, Feb 27 2024, 17:28:07) [MSC v.1916 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          3.0.0
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.3
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             3.1.2
pandas:               2.2.1
pyarrow:              15.0.2
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>```

</details>
@Chuck321123 Chuck321123 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels May 27, 2024
@Chuck321123 Chuck321123 changed the title dt.trunctate is 4x slower in polars compared to pandas dt.trunctate is 3-4x slower in polars compared to pandas May 27, 2024
@cmdlineluser
Copy link
Contributor

I see a similar difference.

This particular operation seems to run faster on the streaming engine.

(df.lazy()
   .with_columns(pl.col("UTC_Time").dt.truncate(f"{Interval}h"))
   .collect(streaming=True)
)

@MarcoGorelli
Copy link
Collaborator

thanks for the report! for a quick look at the code, nothing jumps out at me (like it did in the offset_by one) - time to investigate

@MarcoGorelli MarcoGorelli added performance Performance issues or improvements A-temporal Area: date/time functionality and removed bug Something isn't working needs triage Awaiting prioritization by a maintainer labels May 27, 2024
@Chuck321123 Chuck321123 changed the title dt.trunctate is 3-4x slower in polars compared to pandas dt.truncate is 3-4x slower in polars compared to pandas May 29, 2024
@MarcoGorelli
Copy link
Collaborator

I might have a fix, just testing it out to be sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-temporal Area: date/time functionality performance Performance issues or improvements python Related to Python Polars
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants