Skip to content

Commit

Permalink
[SPARK-42852][SQL] Revert NamedLambdaVariable related changes from Eq…
Browse files Browse the repository at this point in the history
…uivalentExpressions

### What changes were proposed in this pull request?
This PR reverts the follow-up PR of SPARK-41468: apache#39046

### Why are the changes needed?
These changes are not needed and actually might cause performance regression due to preventing higher order function subexpression elimination in `EquivalentExpressions`. Please find related conversation here: apache#40473 (comment)

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Existing UTs.

Closes apache#40475 from peter-toth/SPARK-42852-revert-namedlambdavariable-changes.

Authored-by: Peter Toth <peter.toth@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
peter-toth authored and HyukjinKwon committed Mar 20, 2023
1 parent 1e7e47d commit ce3b03d
Showing 1 changed file with 2 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,9 @@ class EquivalentExpressions {

private def supportedExpression(e: Expression) = {
!e.exists {
// `LambdaVariable` is usually used as a loop variable and `NamedLambdaVariable` is used in
// higher-order functions, which can't be evaluated ahead of the execution.
// `LambdaVariable` is usually used as a loop variable, which can't be evaluated ahead of the
// loop. So we can't evaluate sub-expressions containing `LambdaVariable` at the beginning.
case _: LambdaVariable => true
case _: NamedLambdaVariable => true

// `PlanExpression` wraps query plan. To compare query plans of `PlanExpression` on executor,
// can cause error like NPE.
Expand Down

0 comments on commit ce3b03d

Please sign in to comment.