[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly #5972

pzzs · 2015-05-07T07:04:15Z

SQL

select key from src where 3 in (4, 5);

Before

== Optimized Logical Plan ==
Project [key#12]
 Filter 3 INSET (5,4)
  MetastoreRelation default, src, None

After

== Optimized Logical Plan ==
LocalRelation [key#228], []

merge lastest spark

AmplabJenkins · 2015-05-07T07:07:09Z

Can one of the admins verify this patch?

AmplabJenkins · 2015-05-07T08:02:12Z

Can one of the admins verify this patch?

scwf · 2015-05-07T09:44:37Z

please add a test for this

pzzs · 2015-05-07T12:21:20Z

@scwf

yhuai · 2015-05-07T15:25:49Z

ok to test

AmplabJenkins · 2015-05-07T15:27:12Z

Merged build triggered.

AmplabJenkins · 2015-05-07T15:27:21Z

Merged build started.

SparkQA · 2015-05-07T15:30:04Z

Test build #32117 has started for PR 5972 at commit 24739bd.

SparkQA · 2015-05-07T17:20:10Z

Test build #32117 has finished for PR 5972 at commit 24739bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-08T01:38:05Z

Test build #32174 has started for PR 5972 at commit abe2bbb.

yhuai · 2015-05-08T01:40:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

@@ -46,13 +46,13 @@ object DefaultOptimizer extends Optimizer {
      CombineLimits) ::
    Batch("ConstantFolding", FixedPoint(100),
      NullPropagation,
+      OptimizeIn,


Why move it?

because it more like a transform, i think it should before ConstantFolding @yhuai

All of them are in the batch of "ConstantFolding". So, we do not really need to move them, right?

Not necessary, but looks ok to me, here first do transform from in -> inset, then in ConstantFolding it will call InSet.eval (not In.eval) which should be more efficient.

SparkQA · 2015-05-08T01:56:46Z

Test build #32173 has finished for PR 5972 at commit fa461a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T01:56:49Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T01:56:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32173/
Test FAILed.

AmplabJenkins · 2015-05-08T01:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-08T01:57:20Z

Merged build started.

SparkQA · 2015-05-08T01:59:03Z

Test build #32177 has started for PR 5972 at commit 4c722a2.

SparkQA · 2015-05-08T02:02:50Z

Test build #32174 has finished for PR 5972 at commit abe2bbb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T02:02:54Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T02:02:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32174/
Test FAILed.

SparkQA · 2015-05-08T03:47:28Z

Test build #32177 has finished for PR 5972 at commit 4c722a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T03:47:32Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T03:47:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32177/
Test PASSed.

pzzs · 2015-05-11T09:00:55Z

@marmbrus @yhuai @scwf

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes #5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request #13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master f61210c [Zhongshuai Pei] Merge pull request #9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master 802261c [DoingDone9] Merge pull request #7 from apache/master d00303b [DoingDone9] Merge pull request #6 from apache/master 98b134f [DoingDone9] Merge pull request #5 from apache/master 161cae3 [DoingDone9] Merge pull request #4 from apache/master c87e8b6 [DoingDone9] Merge pull request #3 from apache/master cb1852d [DoingDone9] Merge pull request #2 from apache/master c3f046f [DoingDone9] Merge pull request #1 from apache/master (cherry picked from commit 4b5e1fe) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2015-05-12T02:26:16Z

Thanks, merged to master and 1.4

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes apache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closes #6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by #5972.

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes apache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closes apache#6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by apache#5972.

…" into true or false directly SQL ``` select key from src where 3 in (4, 5); ``` Before ``` == Optimized Logical Plan == Project [key#12] Filter 3 INSET (5,4) MetastoreRelation default, src, None ``` After ``` == Optimized Logical Plan == LocalRelation [key#228], [] ``` Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes apache#5972 from DoingDone9/InToFalse and squashes the following commits: 4c722a2 [Zhongshuai Pei] Update predicates.scala abe2bbb [Zhongshuai Pei] Update Optimizer.scala fa461a5 [Zhongshuai Pei] Update Optimizer.scala e34c28a [Zhongshuai Pei] Update predicates.scala 24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala 35ceb7a [Zhongshuai Pei] Update Optimizer.scala 36c194e [Zhongshuai Pei] Update Optimizer.scala 2e8f6ca [Zhongshuai Pei] Update Optimizer.scala 14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master 802261c [DoingDone9] Merge pull request apache#7 from apache/master d00303b [DoingDone9] Merge pull request apache#6 from apache/master 98b134f [DoingDone9] Merge pull request apache#5 from apache/master 161cae3 [DoingDone9] Merge pull request apache#4 from apache/master c87e8b6 [DoingDone9] Merge pull request apache#3 from apache/master cb1852d [DoingDone9] Merge pull request apache#2 from apache/master c3f046f [DoingDone9] Merge pull request apache#1 from apache/master

Replaced `fs.listFiles` with Hadoop-1 friendly `fs.listStatus` method. Author: Hari Shreedharan <hshreedharan@apache.org> Closes apache#6619 from harishreedharan/evetlog-hadoop-1-fix and squashes the following commits: 6192078 [Hari Shreedharan] [HOTFIX] Fix Hadoop-1 build caused by apache#5972.

pzzs added 13 commits March 2, 2015 10:11

Merge pull request #1 from apache/master

c3f046f

merge lastest spark

Merge pull request #2 from apache/master

cb1852d

merge lastest spark

Merge pull request #3 from apache/master

c87e8b6

merge lastest spark

Merge pull request #4 from apache/master

161cae3

merge lastest spark

Merge pull request #5 from apache/master

98b134f

merge lastest spark

Merge pull request #6 from apache/master

d00303b

merge lastest spark

Merge pull request #7 from apache/master

802261c

merge lastest spark

Merge pull request #8 from apache/master

34b1a9a

merge lastest spark

Merge pull request #9 from apache/master

f61210c

merge lastest spark

Merge pull request #10 from apache/master

f12fa50

merge lastest spark

Merge pull request #12 from apache/master

f03fe7f

merge lastest spark

Merge pull request #13 from apache/master

14952e2

merge lastest spark

Update Optimizer.scala

2e8f6ca

pzzs closed this May 7, 2015

pzzs reopened this May 7, 2015

pzzs changed the title ~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into false directly if not in.~~ [SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly if all elements of list is Literal May 7, 2015

Update Optimizer.scala

36c194e

pzzs added 3 commits May 7, 2015 19:10

Update Optimizer.scala

35ceb7a

Update ConstantFoldingSuite.scala

f4dbf50

Update ConstantFoldingSuite.scala

24739bd

pzzs changed the title ~~[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly if all elements of list is Literal~~ [SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly May 7, 2015

yhuai reviewed May 8, 2015
View reviewed changes

Update predicates.scala

4c722a2

asfgit closed this in 4b5e1fe May 12, 2015

harishreedharan added a commit to harishreedharan/spark that referenced this pull request Jun 3, 2015

[HOTFIX] Fix Hadoop-1 build caused by apache#5972.

a5e6f6a

harishreedharan added a commit to harishreedharan/spark that referenced this pull request Jun 3, 2015

[HOTFIX] Fix Hadoop-1 build caused by apache#5972.

6192078

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly #5972

[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly #5972

pzzs commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

scwf commented May 7, 2015

pzzs commented May 7, 2015

yhuai commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 8, 2015

yhuai May 8, 2015

pzzs May 8, 2015

yhuai May 8, 2015

scwf May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

pzzs commented May 11, 2015

marmbrus commented May 12, 2015

[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly #5972

[SPARK-7437][SQL] Fold "literal in (item1, item2, ..., literal, ...)" into true or false directly #5972

Conversation

pzzs commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

scwf commented May 7, 2015

pzzs commented May 7, 2015

yhuai commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 8, 2015

yhuai May 8, 2015

Choose a reason for hiding this comment

pzzs May 8, 2015

Choose a reason for hiding this comment

yhuai May 8, 2015

Choose a reason for hiding this comment

scwf May 8, 2015

Choose a reason for hiding this comment

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

pzzs commented May 11, 2015

marmbrus commented May 12, 2015