planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join #35325

winoros · 2022-06-13T09:36:14Z

What problem does this PR solve?

Issue Number: close #35323

Problem Summary:

What is changed and how it works?

Add the new hint SEMI_JOIN_REWRITE which is written in the subquery scope.

If the hint is specified, we rewrite the semi join to inner join with aggregatioin just like what tidb_opt_insubq_to_join_and_agg does.

An Example
before

tidb> explain select * from t where exists (select /*+ SEMI_JOIN_REWRITE() */ 1 from t t1 join t t2 where t1.a = t2.a and t1.a = t.a);
+---------------------------------+----------+-----------+---------------+--------------------------------------------+
| id                              | estRows  | task      | access object | operator info                              |
+---------------------------------+----------+-----------+---------------+--------------------------------------------+
| HashJoin_12                     | 8000.00  | root      |               | semi join, equal:[eq(test.t.a, test.t.a)]  |
| ├─HashJoin_15(Build)            | 12500.00 | root      |               | inner join, equal:[eq(test.t.a, test.t.a)] |
| │ ├─TableReader_20(Build)       | 10000.00 | root      |               | data:TableFullScan_19                      |
| │ │ └─TableFullScan_19          | 10000.00 | cop[tikv] | table:t2      | keep order:false, stats:pseudo             |
| │ └─TableReader_18(Probe)       | 10000.00 | root      |               | data:TableFullScan_17                      |
| │   └─TableFullScan_17          | 10000.00 | cop[tikv] | table:t1      | keep order:false, stats:pseudo             |
| └─TableReader_14(Probe)         | 10000.00 | root      |               | data:TableFullScan_13                      |
|   └─TableFullScan_13            | 10000.00 | cop[tikv] | table:t       | keep order:false, stats:pseudo             |
+---------------------------------+----------+-----------+---------------+--------------------------------------------+
8 rows in set, 1 warning (0.01 sec)

After

+-----------------------------------+--------------+-----------+---------------+-------------------------------------------------------+
| id                                | estRows      | task      | access object | operator info                                         |
+-----------------------------------+--------------+-----------+---------------+-------------------------------------------------------+
| HashJoin_16                       | 10000.00     | root      |               | inner join, equal:[eq(test.t.a, test.t.a)]            |
| ├─HashAgg_20(Build)               | 8000.00      | root      |               | group by:test.t.a, funcs:firstrow(test.t.a)->test.t.a |
| │ └─HashJoin_21                   | 100000000.00 | root      |               | CARTESIAN inner join                                  |
| │   ├─TableReader_26(Build)       | 10000.00     | root      |               | data:TableFullScan_25                                 |
| │   │ └─TableFullScan_25          | 10000.00     | cop[tikv] | table:t2      | keep order:false, stats:pseudo                        |
| │   └─TableReader_24(Probe)       | 10000.00     | root      |               | data:TableFullScan_23                                 |
| │     └─TableFullScan_23          | 10000.00     | cop[tikv] | table:t1      | keep order:false, stats:pseudo                        |
| └─TableReader_19(Probe)           | 10000.00     | root      |               | data:TableFullScan_18                                 |
|   └─TableFullScan_18              | 10000.00     | cop[tikv] | table:t       | keep order:false, stats:pseudo                        |
+-----------------------------------+--------------+-----------+---------------+-------------------------------------------------------+
9 rows in set (0.01 sec)

Since Exist semi-join character doesn't care how many rows from the inner side existed, adding distinct here can reduce the size of the second join result set here as we say.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add new optimizer hint `SEMI_JOIN_REWRITE` to control EXISTS's execution behavior.

ti-chi-bot · 2022-06-13T09:36:16Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

AilinKid
time-and-fate

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

winoros

Seems that there're also some aggregations not eliminated

winoros · 2022-06-13T09:38:35Z

planner/core/testdata/plan_suite_unexported_out.json

+      "Join{DataScan(t)->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->Sel([lt(test.t.a, 1)])->Projection",
+      "Join{Join{DataScan(t)->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->Projection",
+      "Join{Join{DataScan(t)->DataScan(x)}(test.t.a,test.t.a)->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->Projection",
+      "Join{Join{DataScan(t)->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->DataScan(x)->Aggr(firstrow(test.t.a))}(test.t.a,test.t.a)->Projection->Projection",


The two semi-joins should all be able rewriting.
Needs further checking.

winoros · 2022-06-13T09:40:17Z

The hint cannot be captured by the SPM(since the behavior of the hint is not enabled by default. So it's okay that it's not captured currently).

planner/core/rule_semi_join_rewrite.go

time-and-fate · 2022-06-20T13:46:53Z

planner/core/logical_plan_builder.go

@@ -3626,6 +3628,12 @@ func (b *PlanBuilder) pushTableHints(hints []*ast.TableOptimizerHint, currentLev
 				leadingJoinOrder = append(leadingJoinOrder, tableNames2HintTableInfo(b.ctx, hint.HintName.L, hint.Tables, b.hintProcessor, currentLevel)...)
 			}
 			leadingHintCnt++
+		case HintSemiJoinRewrite:
+			if !b.checkSemiJoinHint {
+				b.ctx.GetSessionVars().StmtCtx.AppendWarning(ErrInternal.GenWithStack("The SEMI_JOIN_REWRITE hint is not used correctly, maybe it's not in a subquery or the subquery is not EXSITS clause."))


Suggested change

b.ctx.GetSessionVars().StmtCtx.AppendWarning(ErrInternal.GenWithStack("The SEMI_JOIN_REWRITE hint is not used correctly, maybe it's not in a subquery or the subquery is not EXSITS clause."))

b.ctx.GetSessionVars().StmtCtx.AppendWarning(ErrInternal.GenWithStack("The SEMI_JOIN_REWRITE hint is not used correctly, maybe it's not in a subquery or the subquery is not EXISTS clause."))

planner/core/rule_semi_join_rewrite.go

time-and-fate · 2022-06-27T09:09:09Z

planner/core/rule_semi_join_rewrite.go

+	subAgg := LogicalAggregation{
+		AggFuncs:     make([]*aggregation.AggFuncDesc, 0, len(join.EqualConditions)),
+		GroupByItems: make([]expression.Expression, 0, len(join.EqualConditions)),
+	}.Init(p.SCtx(), p.Children()[1].SelectBlockOffset())
+
+	aggOutputCols := make([]*expression.Column, 0, len(join.EqualConditions))
+	for i := range join.EqualConditions {
+		innerCol := join.EqualConditions[i].GetArgs()[1].(*expression.Column)
+		firstRow, err := aggregation.NewAggFuncDesc(join.SCtx(), ast.AggFuncFirstRow, []expression.Expression{innerCol}, false)
+		if err != nil {
+			return nil, err
+		}
+		subAgg.AggFuncs = append(subAgg.AggFuncs, firstRow)
+		subAgg.GroupByItems = append(subAgg.GroupByItems, innerCol)
+		aggOutputCols = append(aggOutputCols, innerCol)
+	}
+	subAgg.SetChildren(innerChild)
+	subAgg.SetSchema(expression.NewSchema(aggOutputCols...))
+	subAgg.buildSelfKeyInfo(subAgg.Schema())


Can we directly use buildDistinct() like we do in handleInSubquery()?

In buildDistinct it also deals with the hints or the optFlag which is not needed after the plan building phase.

I think we can keep the current codes since it's a very simple rewrite.

AilinKid

Rest LGTM

AilinKid · 2022-06-28T10:28:21Z

planner/core/rule_semi_join_rewrite.go

+	}.Init(p.SCtx(), p.Children()[1].SelectBlockOffset())
+
+	aggOutputCols := make([]*expression.Column, 0, len(join.EqualConditions))
+	for i := range join.EqualConditions {


we can reuse the buildDistinct here like InSubquery？seems the last &columnPruner{} can reduce the unused column if needed.

planner/core/rule_semi_join_rewrite.go

planner/core/testdata/integration_suite_out.json

…o semi-join-rewrite-with-hint

AilinKid · 2022-06-29T09:33:06Z

Care for the license header from
"ERROR the following files don't have a valid license header: planner/core/rule_semi_join_rewrite.go "

Rest LGTM

…o semi-join-rewrite-with-hint

time-and-fate

Rest LGTM.
And please make it pass the CI.

time-and-fate · 2022-07-01T07:55:04Z

planner/core/expression_rewriter.go

@@ -315,7 +315,9 @@ func (er *expressionRewriter) constructBinaryOpFunction(l expression.Expression,
 	}
 }

-func (er *expressionRewriter) buildSubquery(ctx context.Context, subq *ast.SubqueryExpr) (LogicalPlan, error) {
+// buildSubquery translates the subquery ast to plan.
+// Currently, only the IN/EXIST can apply the rewrite hint(rewrite the semi join to inner join with aggregation).


I think this line is a bit misleading.
The hint is applicable for EXISTS only. The rewrite for IN is not controlled by the hint.

…o semi-join-rewrite-with-hint

winoros · 2022-07-12T08:26:19Z

/run-unit-test

winoros · 2022-07-12T08:33:33Z

/run-check_dev

sre-bot · 2022-07-12T08:33:37Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/ab3778347e3efbe56896d09741e4c6000f16c75a

winoros · 2022-07-13T08:03:52Z

/run-mysql-test

winoros · 2022-07-14T06:36:08Z

/merge

ti-chi-bot · 2022-07-14T06:36:12Z

This pull request has been accepted and is ready to merge.

Commit hash: 0397da9

winoros · 2022-07-14T07:46:19Z

/run-check_dev_2

sre-bot · 2022-07-14T08:35:10Z

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci/integration-cdc-test	🔴 failed 1, success 35, total 36	38 min	Existing failure
idc-jenkins-ci-tidb/integration-common-test	✅ all 11 tests passed	32 min	Fixed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	18 min	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	10 min	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	9 min 31 sec	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	8 min 50 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	8 min 29 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	8 min 9 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	4 min 22 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

ti-srebot · 2022-08-22T14:14:01Z

cherry pick to release-6.1 in PR #37283

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

…#37283) close #35323

winoros added 2 commits June 13, 2022 16:45

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join

5b190ef

add tests

5f004c3

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 13, 2022

winoros added sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 13, 2022

winoros requested review from time-and-fate and AilinKid June 13, 2022 09:36

winoros commented Jun 13, 2022

View reviewed changes

add the wrongly not added file

a8467b0

ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 13, 2022

Merge branch 'master' into semi-join-rewrite-with-hint

088360c

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2022

time-and-fate reviewed Jun 20, 2022

View reviewed changes

time-and-fate reviewed Jun 27, 2022

View reviewed changes

Merge branch 'master' into semi-join-rewrite-with-hint

6e73b42

AilinKid reviewed Jun 28, 2022

View reviewed changes

winoros added 2 commits June 29, 2022 17:15

fix the bug found by AilinKid

ba43f75

Merge remote-tracking branch 'yiding/semi-join-rewrite-with-hint' int…

4b82ca3

…o semi-join-rewrite-with-hint

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 29, 2022

Merge branch 'master' into semi-join-rewrite-with-hint

71ff868

winoros and others added 5 commits June 29, 2022 20:28

add tests

c563356

Merge remote-tracking branch 'yiding/semi-join-rewrite-with-hint' int…

78de4de

…o semi-join-rewrite-with-hint

fix license header

9670a0f

fix typo

332adef

Merge branch 'master' into semi-join-rewrite-with-hint

e5ee1ae

AilinKid approved these changes Jun 30, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 30, 2022

time-and-fate approved these changes Jul 1, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 1, 2022

AilinKid and others added 4 commits July 7, 2022 11:39

Merge branch 'master' into semi-join-rewrite-with-hint

ded3884

Merge branch 'master' into semi-join-rewrite-with-hint

796ecbb

fix test

dfabf99

Merge remote-tracking branch 'yiding/semi-join-rewrite-with-hint' int…

3bc72a8

…o semi-join-rewrite-with-hint

fix comment

0397da9

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 14, 2022

ti-chi-bot added 2 commits July 14, 2022 14:36

Merge branch 'master' into semi-join-rewrite-with-hint

385184e

Merge branch 'master' into semi-join-rewrite-with-hint

ab37783

ti-chi-bot merged commit b4cd14d into pingcap:master Jul 14, 2022

winoros deleted the semi-join-rewrite-with-hint branch July 14, 2022 07:49

chrysan added the needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. label Aug 22, 2022

ti-srebot mentioned this pull request Aug 22, 2022

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join (#35325) #37283

Merged

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Aug 22, 2022

cherry pick pingcap#35325 to release-6.1

24a9265

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-chi-bot pushed a commit that referenced this pull request Aug 23, 2022

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join (#35325) (…

bf0e7a1

…#37283) close #35323

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join #35325

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join #35325

winoros commented Jun 13, 2022 •

edited by AilinKid

Loading

ti-chi-bot commented Jun 13, 2022 •

edited

Loading

winoros left a comment

winoros Jun 13, 2022

winoros commented Jun 13, 2022

time-and-fate Jun 20, 2022

time-and-fate Jun 27, 2022

winoros Jun 29, 2022

AilinKid left a comment

AilinKid Jun 28, 2022

AilinKid commented Jun 29, 2022

time-and-fate left a comment

time-and-fate Jul 1, 2022

winoros commented Jul 12, 2022

winoros commented Jul 12, 2022

sre-bot commented Jul 12, 2022 •

edited

Loading

winoros commented Jul 13, 2022

winoros commented Jul 14, 2022

ti-chi-bot commented Jul 14, 2022

winoros commented Jul 14, 2022

sre-bot commented Jul 14, 2022

ti-srebot commented Aug 22, 2022

	b.ctx.GetSessionVars().StmtCtx.AppendWarning(ErrInternal.GenWithStack("The SEMI_JOIN_REWRITE hint is not used correctly, maybe it's not in a subquery or the subquery is not EXSITS clause."))
	b.ctx.GetSessionVars().StmtCtx.AppendWarning(ErrInternal.GenWithStack("The SEMI_JOIN_REWRITE hint is not used correctly, maybe it's not in a subquery or the subquery is not EXISTS clause."))

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join #35325

planner: use SEMI_JOIN_REWRITE hint to rewrite the semi join #35325

Conversation

winoros commented Jun 13, 2022 • edited by AilinKid Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 13, 2022 • edited Loading

winoros left a comment

Choose a reason for hiding this comment

winoros Jun 13, 2022

Choose a reason for hiding this comment

winoros commented Jun 13, 2022

time-and-fate Jun 20, 2022

Choose a reason for hiding this comment

time-and-fate Jun 27, 2022

Choose a reason for hiding this comment

winoros Jun 29, 2022

Choose a reason for hiding this comment

AilinKid left a comment

Choose a reason for hiding this comment

AilinKid Jun 28, 2022

Choose a reason for hiding this comment

AilinKid commented Jun 29, 2022

time-and-fate left a comment

Choose a reason for hiding this comment

time-and-fate Jul 1, 2022

Choose a reason for hiding this comment

winoros commented Jul 12, 2022

winoros commented Jul 12, 2022

sre-bot commented Jul 12, 2022 • edited Loading

winoros commented Jul 13, 2022

winoros commented Jul 14, 2022

ti-chi-bot commented Jul 14, 2022

winoros commented Jul 14, 2022

sre-bot commented Jul 14, 2022

TiDB MergeCI notify

ti-srebot commented Aug 22, 2022

winoros commented Jun 13, 2022 •

edited by AilinKid

Loading

ti-chi-bot commented Jun 13, 2022 •

edited

Loading

sre-bot commented Jul 12, 2022 •

edited

Loading