[SPARK-1495][SQL]add support for left semi join #837

adrian-wang · 2014-05-20T09:35:12Z

Just submit another solution for #395

adrian-wang · 2014-05-20T09:37:10Z

This is a solution with #418 from @marmbrus .

AmplabJenkins · 2014-05-20T09:37:58Z

Can one of the admins verify this patch?

rxin · 2014-05-21T08:27:48Z

Jenkins, add to whitelist.

AmplabJenkins · 2014-05-21T08:28:00Z

Merged build triggered.

AmplabJenkins · 2014-05-21T08:28:09Z

Merged build started.

AmplabJenkins · 2014-05-21T08:30:04Z

Merged build finished.

AmplabJenkins · 2014-05-21T08:30:04Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15115/

AmplabJenkins · 2014-05-21T08:47:57Z

Merged build triggered.

AmplabJenkins · 2014-05-21T08:48:04Z

Merged build started.

AmplabJenkins · 2014-05-21T10:19:08Z

Merged build finished.

AmplabJenkins · 2014-05-21T10:19:08Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15116/

AmplabJenkins · 2014-05-21T15:52:58Z

Merged build triggered.

AmplabJenkins · 2014-05-21T15:53:06Z

Merged build started.

AmplabJenkins · 2014-05-21T17:24:14Z

Merged build finished.

AmplabJenkins · 2014-05-21T17:24:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15121/

marmbrus · 2014-05-31T18:34:29Z

sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala

@@ -144,6 +144,150 @@ case class HashJoin(
 * :: DeveloperApi ::
 */
 @DeveloperApi
+case class LeftSemiJoinHash(
+                     leftKeys: Seq[Expression],


Intent only 4 spaces here.

marmbrus · 2014-05-31T18:55:51Z

This is getting closer. Thanks for working on it!

marmbrus · 2014-05-31T18:56:57Z

Can you add "[SPARK-1495][SQL]" to the PR title?

AmplabJenkins · 2014-06-03T05:57:58Z

Merged build triggered.

AmplabJenkins · 2014-06-03T05:58:07Z

Merged build started.

adrian-wang · 2014-06-03T06:01:40Z

Hi Michael, when I was adding the Scala doc, I realized that if the join is not calculated in LeftSemiJoinHash, then it simply means there's no join keys for the left semi join. Then if I pushed down those predicates and conditions(all of them can be pushed down), I only need to verify the if right table size is null here, to decide whether to output the left table. So LeftSemiJoinBNL would be very much excessive. Am I right?

AmplabJenkins · 2014-06-03T07:12:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-03T07:12:46Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15374/

marmbrus · 2014-06-07T17:08:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala

@@ -119,6 +119,11 @@ object HashFilteredJoin extends Logging with PredicateHelper {
    case FilteredOperation(predicates, join @ Join(left, right, Inner, condition)) =>
      logger.debug(s"Considering hash inner join on: ${predicates ++ condition}")
      splitPredicates(predicates ++ condition, join)
+    // All predicates can be evaluated for left semi join (those that are in the WHERE
+    // clause can only from left table, so they can all be pushed down.)


I think in general we should avoid making too many assumptions in the planner about what optimizations have occurred. For example, in the future we might avoid pushing down predicates that are very expensive to evaluate as it might be cheaper to run them on an already filtered set of data. However, in the case of LEFT SEMI JOIN, I think it is actually okay to push all evaluation into the join condition, even if they only refer to the left table. Is that true?

Yes, I think LEFT SEMI JOIN would not suffer by pushing down predicates.

marmbrus · 2014-06-07T19:24:28Z

(I deleted my earlier comment because I found a mistake)

I think this is looking pretty good, but we should at least add one test for the Broadcast Nested Loop version. Here's a PR against your branch that does that: adrian-wang#1

Add a test.

AmplabJenkins · 2014-06-08T06:37:50Z

Merged build triggered.

AmplabJenkins · 2014-06-08T06:53:51Z

Merged build started.

AmplabJenkins · 2014-06-08T08:09:10Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-08T08:09:10Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15540/

Just submit another solution for #395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes #837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request #1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join (cherry picked from commit 0cf6002) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2014-06-09T18:34:59Z

Thanks! I've merged this into 1.0 and master.

Some improvement for PR #837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes #1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

Some improvement for PR #837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes #1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com> (cherry picked from commit ce6deb1) Signed-off-by: Michael Armbrust <michael@databricks.com>

Just submit another solution for apache#395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes apache#837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request apache#1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join

Some improvement for PR apache#837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes apache#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

Just submit another solution for apache#395 Author: Daoyuan <daoyuan.wang@intel.com> Author: Michael Armbrust <michael@databricks.com> Author: Daoyuan Wang <daoyuan.wang@intel.com> Closes apache#837 from adrian-wang/left-semi-join-support and squashes the following commits: d39cd12 [Daoyuan Wang] Merge pull request apache#1 from marmbrus/pr/837 6713c09 [Michael Armbrust] Better debugging for failed query tests. 035b73e [Michael Armbrust] Add test for left semi that can't be done with a hash join. 5ec6fa4 [Michael Armbrust] Add left semi to SQL Parser. 4c726e5 [Daoyuan] improvement according to Michael 8d4a121 [Daoyuan] add golden files for leftsemijoin 83a3c8a [Daoyuan] scala style fix 14cff80 [Daoyuan] add support for left semi join

Some improvement for PR apache#837, add another case to white list and use `filter` to build result iterator. Author: Daoyuan <daoyuan.wang@intel.com> Closes apache#1049 from adrian-wang/clean-LeftSemiJoinHash and squashes the following commits: b314d5a [Daoyuan] change hashSet name 27579a9 [Daoyuan] add semijoin to white list and use filter to create new iterator in LeftSemiJoinBNL Signed-off-by: Michael Armbrust <michael@databricks.com>

…tastore version (apache#837)

add support for left semi join

14cff80

adrian-wang mentioned this pull request May 21, 2014

support leftsemijoin for sparkSQL #395

Closed

scala style fix

83a3c8a

add golden files for leftsemijoin

8d4a121

marmbrus reviewed May 31, 2014
View reviewed changes

adrian-wang changed the title ~~add support for left semi join~~ [SPARK-1495][SQL]add support for left semi join Jun 2, 2014

improvement according to Michael

4c726e5

marmbrus reviewed Jun 7, 2014
View reviewed changes

marmbrus added 3 commits June 7, 2014 12:22

Add left semi to SQL Parser.

5ec6fa4

Add test for left semi that can't be done with a hash join.

035b73e

Better debugging for failed query tests.

6713c09

Merge pull request #1 from marmbrus/pr/837

d39cd12

Add a test.

asfgit closed this in 0cf6002 Jun 9, 2014

adrian-wang mentioned this pull request Jun 11, 2014

Clean left semi join hash #1049

Closed

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-891] Thrift server start fails due to unsupported Hive Me…

ac5e16b

…tastore version (apache#837)

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-891] Thrift server start fails due to unsupported Hive Me…

6f189e1

…tastore version (apache#837)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-1495][SQL]add support for left semi join #837

[SPARK-1495][SQL]add support for left semi join #837

adrian-wang commented May 20, 2014

adrian-wang commented May 20, 2014

AmplabJenkins commented May 20, 2014

rxin commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

marmbrus May 31, 2014

marmbrus commented May 31, 2014

marmbrus commented May 31, 2014

AmplabJenkins commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

adrian-wang commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

marmbrus Jun 7, 2014

adrian-wang Jun 8, 2014

marmbrus commented Jun 7, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

marmbrus commented Jun 9, 2014

[SPARK-1495][SQL]add support for left semi join #837

[SPARK-1495][SQL]add support for left semi join #837

Conversation

adrian-wang commented May 20, 2014

adrian-wang commented May 20, 2014

AmplabJenkins commented May 20, 2014

rxin commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

AmplabJenkins commented May 21, 2014

marmbrus May 31, 2014

Choose a reason for hiding this comment

marmbrus commented May 31, 2014

marmbrus commented May 31, 2014

AmplabJenkins commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

adrian-wang commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

AmplabJenkins commented Jun 3, 2014

marmbrus Jun 7, 2014

Choose a reason for hiding this comment

adrian-wang Jun 8, 2014

Choose a reason for hiding this comment

marmbrus commented Jun 7, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

AmplabJenkins commented Jun 8, 2014

marmbrus commented Jun 9, 2014