Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter is not pushed down through joins #18619

Open
2 tasks done
coastalwhite opened this issue Sep 9, 2024 · 2 comments
Open
2 tasks done

Filter is not pushed down through joins #18619

coastalwhite opened this issue Sep 9, 2024 · 2 comments
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@coastalwhite
Copy link
Collaborator

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

left = pl.LazyFrame({ 'a': [1, 2, 3], 'b': [2, 2, 2] })
right = pl.LazyFrame({ 'x': [7, 1, 2] })

df = left.join(right, how = 'cross').filter(pl.col.a == pl.col.b)

print(df.explain())

This outputs:

FILTER [(col("a")) == (col("b"))] FROM
  CROSS JOIN:
  LEFT PLAN ON: []
    DF ["a", "b"]; PROJECT */2 COLUMNS; SELECTION: None
  RIGHT PLAN ON: []
    DF ["x"]; PROJECT */1 COLUMNS; SELECTION: None
  END CROSS JOIN

Even though it should be able to optimize it to:

CROSS JOIN:
LEFT PLAN ON: []
  DF ["a", "b"]; PROJECT */2 COLUMNS; SELECTION: [(col("a")) == (col("b"))]
RIGHT PLAN ON: []
  DF ["x"]; PROJECT */1 COLUMNS; SELECTION: None
END CROSS JOIN

Log output

No response

Issue description

Filters are not pushed down through joins, even when that is possible.

Expected behavior

Filters are pushed down.

Installed versions

Replace this line with the output of pl.show_versions(). Leave the backticks in place.
@coastalwhite coastalwhite added bug Something isn't working python Related to Python Polars needs triage Awaiting prioritization by a maintainer labels Sep 9, 2024
@ritchie46
Copy link
Member

Ouch.. Can you take this one?

@mcrumiller
Copy link
Contributor

This is true even for left/inner joins too, not just cross:

left = pl.LazyFrame({"a": [1, 2, 3], "b": [1, 2, 9]})
right = pl.LazyFrame({"a": [1, 2, 3], "c": [4, 5, 6]})
out = left.join(right, how="left", on="a").filter(col("a") == col("b"))
print(out.explain())
FILTER [(col("a")) == (col("b"))] FROM
  LEFT JOIN:
  LEFT PLAN ON: [col("a")]
    DF ["a", "b"]; PROJECT */2 COLUMNS; SELECTION: None
  RIGHT PLAN ON: [col("a")]
    DF ["a", "c"]; PROJECT */2 COLUMNS; SELECTION: None
  END LEFT JOIN

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

3 participants