plan, expression: support `is null` in hash partition pruning #17281

imtbkcat · 2020-05-19T05:40:07Z

What problem does this PR solve?

Problem Summary:

SQL like select * from t where a is null will produce a full table scan plan in hash partition. For example:

create table t (a int, b int) partition by hash(a + b) partitions 10;
explain select * from t where a is null and b = 2;

This SQL will produce following result:

MySQL [test]> explain select * from t where a is null and b = 2;
+------------------------------+----------+-----------+-----------------------+-----------------------------------+
| id                           | estRows  | task      | access object         | operator info                     |
+------------------------------+----------+-----------+-----------------------+-----------------------------------+
| Union_16                     | 0.10     | root      |                       |                                   |
| ├─TableReader_19             | 0.01     | root      |                       | data:Selection_18                 |
| │ └─Selection_18             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_17       | 10000.00 | cop[tikv] | table:t, partition:p0 | keep order:false, stats:pseudo    |
| ├─TableReader_22             | 0.01     | root      |                       | data:Selection_21                 |
| │ └─Selection_21             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_20       | 10000.00 | cop[tikv] | table:t, partition:p1 | keep order:false, stats:pseudo    |
| ├─TableReader_25             | 0.01     | root      |                       | data:Selection_24                 |
| │ └─Selection_24             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_23       | 10000.00 | cop[tikv] | table:t, partition:p2 | keep order:false, stats:pseudo    |
| ├─TableReader_28             | 0.01     | root      |                       | data:Selection_27                 |
| │ └─Selection_27             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_26       | 10000.00 | cop[tikv] | table:t, partition:p3 | keep order:false, stats:pseudo    |
| ├─TableReader_31             | 0.01     | root      |                       | data:Selection_30                 |
| │ └─Selection_30             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_29       | 10000.00 | cop[tikv] | table:t, partition:p4 | keep order:false, stats:pseudo    |
| ├─TableReader_34             | 0.01     | root      |                       | data:Selection_33                 |
| │ └─Selection_33             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_32       | 10000.00 | cop[tikv] | table:t, partition:p5 | keep order:false, stats:pseudo    |
| ├─TableReader_37             | 0.01     | root      |                       | data:Selection_36                 |
| │ └─Selection_36             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_35       | 10000.00 | cop[tikv] | table:t, partition:p6 | keep order:false, stats:pseudo    |
| ├─TableReader_40             | 0.01     | root      |                       | data:Selection_39                 |
| │ └─Selection_39             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_38       | 10000.00 | cop[tikv] | table:t, partition:p7 | keep order:false, stats:pseudo    |
| ├─TableReader_43             | 0.01     | root      |                       | data:Selection_42                 |
| │ └─Selection_42             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
| │   └─TableFullScan_41       | 10000.00 | cop[tikv] | table:t, partition:p8 | keep order:false, stats:pseudo    |
| └─TableReader_46             | 0.01     | root      |                       | data:Selection_45                 |
|   └─Selection_45             | 0.01     | cop[tikv] |                       | eq(test.t.b, 2), isnull(test.t.a) |
|     └─TableFullScan_44       | 10000.00 | cop[tikv] | table:t, partition:p9 | keep order:false, stats:pseudo    |
+------------------------------+----------+-----------+-----------------------+-----------------------------------+

What is changed and how it works?

Proposal: xxx

What's Changed:

During partition pruning, if a column is equal to NULL, all calculation relating to it will produce null result and set a NULL flag.

How it Works:

If pruning result with a NULL flag, it will be point to p0.

Related changes

Need to cherry-pick to the release branch

Check List

Tests

Unit test

Side effects

No

Release note

planner: support is null filter condition in hash partition pruning

codecov · 2020-05-19T05:47:38Z

Codecov Report

Merging #17281 into master will decrease coverage by 0.0001%.
The diff coverage is 83.8709%.

@@               Coverage Diff                @@
##             master     #17281        +/-   ##
================================================
- Coverage   79.7957%   79.7956%   -0.0002%     
================================================
  Files           520        517         -3     
  Lines        139624     139534        -90     
================================================
- Hits         111414     111342        -72     
+ Misses        19238      19230         -8     
+ Partials       8972       8962        -10

XuHuaiyu · 2020-05-19T06:15:40Z

expression/partition_pruner.go

 	switch pi := piExpr.(type) {
 	case *ScalarFunction:
 		if pi.FuncName.L == ast.Plus || pi.FuncName.L == ast.Minus || pi.FuncName.L == ast.Mul || pi.FuncName.L == ast.Div {
 			left, right := pi.GetArgs()[0], pi.GetArgs()[1]
-			leftVal, ok, isNil := p.tryEvalPartitionExpr(left)
+			leftVal, ok, isNull := p.tryEvalPartitionExpr(left)
 			if !ok {


if !ok || isNull ?

expression/partition_pruner.go

tiancaiamao · 2020-05-19T15:28:34Z

LGTM
Please fix CI

imtbkcat · 2020-05-20T02:47:09Z

@tiancaiamao @XuHuaiyu PTAL

XuHuaiyu · 2020-05-20T03:42:01Z

expression/partition_pruner.go

 				if err != nil {
 					return 0, false, false
+				} else if isNull {
+					return 0, true, true
 				}
 				return ret, true, false


return ret, true, isNull is enough?

yes, for expression like (year(null) + 2) or year(null), MySQL and TiDB both produce NULL

what I mean is that:
we can remove line 140~141, and change line 143 to return ret, true, isNull

XuHuaiyu · 2020-05-20T03:48:34Z

expression/testdata/partition_pruner_in.json

      "explain select * from t4 where d = '2019-10-07 10:40:00' and a = 1",
-      "explain select * from t5 where d = '2019-10-07'"
+      "explain select * from t5 where d = '2019-10-07'",
+      "explain select * from t6 where a is null",


add a case for explain select * from t6 where b is null

XuHuaiyu

LGTM

XuHuaiyu · 2020-05-20T05:55:41Z

/merge

sre-bot · 2020-05-20T05:58:17Z

/run-all-tests

sre-bot · 2020-05-20T06:03:24Z

@imtbkcat merge failed.

imtbkcat · 2020-05-20T06:24:56Z

/merge

sre-bot · 2020-05-20T06:27:16Z

/run-all-tests

XuHuaiyu · 2020-05-20T07:43:45Z

/merge

sre-bot · 2020-05-20T07:43:48Z

Your auto merge job has been accepted, waiting for:

17195

sre-bot · 2020-05-20T07:50:46Z

/run-all-tests

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot · 2020-05-20T07:55:50Z

cherry pick to release-3.0 in PR #17308

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot · 2020-05-20T07:56:50Z

cherry pick to release-3.1 in PR #17309

…p#17281)

sre-bot · 2020-05-20T07:57:46Z

cherry pick to release-4.0 in PR #17310

#17310)

#17308) * cherry pick #17281 to release-3.0 Signed-off-by: sre-bot <sre-bot@pingcap.com> * fix conflict * fix ci Co-authored-by: Lingyu Song <songlingyu@pingcap.com> Co-authored-by: tiancaiamao <tiancaiamao@gmail.com> Co-authored-by: ti-srebot <66930949+ti-srebot@users.noreply.github.com>

imtbkcat added 3 commits May 19, 2020 12:33

support isnull for hash partition

5f04d0f

fix test

65c4b34

add test

612ee87

imtbkcat added type/bugfix This PR fixes a bug. sig/planner SIG: Planner component/expression needs-cherry-pick-3.0 labels May 19, 2020

imtbkcat requested review from tiancaiamao and XuHuaiyu May 19, 2020 05:40

imtbkcat requested a review from a team as a code owner May 19, 2020 05:40

ghost removed their request for review May 19, 2020 05:40

XuHuaiyu reviewed May 19, 2020

View reviewed changes

imtbkcat added 2 commits May 19, 2020 18:55

address comment

3fdedce

address comment

4029af6

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label May 19, 2020

fix ci

1e341d5

XuHuaiyu reviewed May 20, 2020

View reviewed changes

imtbkcat added 2 commits May 20, 2020 13:39

address comment

e79d57c

address comment

7bde29e

XuHuaiyu approved these changes May 20, 2020

View reviewed changes

sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 20, 2020

XuHuaiyu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 20, 2020

Merge branch 'master' into hashPartitionNull

0de7166

Merge branch 'master' into hashPartitionNull

bf3d511

Merge branch 'master' into hashPartitionNull

8308a1c

sre-bot merged commit f17da4a into pingcap:master May 20, 2020

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request May 20, 2020

cherry pick pingcap#17281 to release-3.0

492d433

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot mentioned this pull request May 20, 2020

plan, expression: support is null in hash partition pruning (#17281) #17308

Merged

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request May 20, 2020

cherry pick pingcap#17281 to release-3.1

efd2045

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot mentioned this pull request May 20, 2020

plan, expression: support is null in hash partition pruning (#17281) #17309

Closed

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request May 20, 2020

plan, expression: support is null in hash partition pruning (pingca…

1e872b1

…p#17281)

sre-bot mentioned this pull request May 20, 2020

plan, expression: support is null in hash partition pruning (#17281) #17310

Merged

sre-bot added a commit that referenced this pull request May 20, 2020

plan, expression: support is null in hash partition pruning (#17281) (

eabc946

#17310)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plan, expression: support `is null` in hash partition pruning #17281

plan, expression: support `is null` in hash partition pruning #17281

imtbkcat commented May 19, 2020

codecov bot commented May 19, 2020 •

edited

Loading

XuHuaiyu May 19, 2020

tiancaiamao commented May 19, 2020

imtbkcat commented May 20, 2020

XuHuaiyu May 20, 2020

imtbkcat May 20, 2020

XuHuaiyu May 20, 2020

XuHuaiyu May 20, 2020

imtbkcat May 20, 2020

XuHuaiyu left a comment

XuHuaiyu commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

imtbkcat commented May 20, 2020

sre-bot commented May 20, 2020

XuHuaiyu commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

plan, expression: support is null in hash partition pruning #17281

plan, expression: support is null in hash partition pruning #17281

Conversation

imtbkcat commented May 19, 2020

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

codecov bot commented May 19, 2020 • edited Loading

Codecov Report

XuHuaiyu May 19, 2020

Choose a reason for hiding this comment

tiancaiamao commented May 19, 2020

imtbkcat commented May 20, 2020

XuHuaiyu May 20, 2020

Choose a reason for hiding this comment

imtbkcat May 20, 2020

Choose a reason for hiding this comment

XuHuaiyu May 20, 2020

Choose a reason for hiding this comment

XuHuaiyu May 20, 2020

Choose a reason for hiding this comment

imtbkcat May 20, 2020

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

XuHuaiyu commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

imtbkcat commented May 20, 2020

sre-bot commented May 20, 2020

XuHuaiyu commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

sre-bot commented May 20, 2020

plan, expression: support `is null` in hash partition pruning #17281

plan, expression: support `is null` in hash partition pruning #17281

codecov bot commented May 19, 2020 •

edited

Loading