Skip to content

Commit

Permalink
[SPARK-41468][SQL][FOLLOWUP] Handle NamedLambdaVariables in Equivalen…
Browse files Browse the repository at this point in the history
…tExpressions

### What changes were proposed in this pull request?
This is a follow-up PR to apache#39010 to handle `NamedLambdaVariable`s too.

### Why are the changes needed?
To avoid possible issues with higer-order functions.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Existing UTs.

Closes apache#39046 from peter-toth/SPARK-41468-fix-planexpressions-in-equivalentexpressions-follow-up.

Authored-by: Peter Toth <peter.toth@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
peter-toth authored and beliefer committed Dec 18, 2022
1 parent 06634d4 commit e923849
Showing 1 changed file with 3 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,10 @@ class EquivalentExpressions {

private def supportedExpression(e: Expression) = {
!e.exists {
// `LambdaVariable` is usually used as a loop variable, which can't be evaluated ahead of the
// loop. So we can't evaluate sub-expressions containing `LambdaVariable` at the beginning.
// `LambdaVariable` is usually used as a loop variable and `NamedLambdaVariable` is used in
// higher-order functions, which can't be evaluated ahead of the execution.
case _: LambdaVariable => true
case _: NamedLambdaVariable => true

// `PlanExpression` wraps query plan. To compare query plans of `PlanExpression` on executor,
// can cause error like NPE.
Expand Down

0 comments on commit e923849

Please sign in to comment.