Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Vectorized nested loop join for in-memory engine #20495

Merged
merged 15 commits into from
Dec 31, 2024
Merged

perf: Vectorized nested loop join for in-memory engine #20495

merged 15 commits into from
Dec 31, 2024

Conversation

ritchie46
Copy link
Member

@ritchie46 ritchie46 commented Dec 29, 2024

Cross-join + filter and ie_join predicates that aren't equi join or ie-join can be filtered during the cross join operation, thereby often drastically reducing maximum memory usage.

supersedes #20329

@ritchie46 ritchie46 changed the title perf: vectorized nested loop join perf: Vectorized nested loop join for in-memory engine Dec 30, 2024
@ritchie46 ritchie46 marked this pull request as ready for review December 31, 2024 10:26
@ritchie46 ritchie46 requested a review from wence- as a code owner December 31, 2024 10:26
@ritchie46 ritchie46 added P-high Priority: high highlight Highlight this PR in the changelog and removed P-high Priority: high title needs formatting labels Dec 31, 2024
@ritchie46 ritchie46 merged commit 4c14e70 into main Dec 31, 2024
48 of 52 checks passed
@ritchie46 ritchie46 deleted the cross branch December 31, 2024 13:03
Copy link

codecov bot commented Dec 31, 2024

Codecov Report

Attention: Patch coverage is 88.25503% with 35 lines in your changes missing coverage. Please review.

Project coverage is 79.04%. Comparing base (b430f64) to head (b4c3ee9).
Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-ops/src/frame/join/args.rs 35.00% 13 Missing ⚠️
...rates/polars-python/src/lazyframe/visitor/nodes.rs 0.00% 11 Missing ⚠️
crates/polars-ops/src/frame/join/cross_join.rs 95.29% 4 Missing ⚠️
crates/polars-mem-engine/src/executors/join.rs 75.00% 1 Missing ⚠️
crates/polars-mem-engine/src/executors/sort.rs 0.00% 1 Missing ⚠️
crates/polars-ops/src/frame/join/mod.rs 97.56% 1 Missing ⚠️
...s/polars-pipe/src/executors/operators/reproject.rs 0.00% 1 Missing ⚠️
crates/polars-plan/src/dsl/options.rs 94.11% 1 Missing ⚠️
.../polars-plan/src/plans/optimizer/collapse_joins.rs 91.66% 1 Missing ⚠️
...lan/src/plans/optimizer/predicate_pushdown/join.rs 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20495      +/-   ##
==========================================
+ Coverage   79.02%   79.04%   +0.01%     
==========================================
  Files        1563     1563              
  Lines      220594   220779     +185     
  Branches     2502     2502              
==========================================
+ Hits       174327   174510     +183     
- Misses      45693    45695       +2     
  Partials      574      574              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
highlight Highlight this PR in the changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant