*: add explicit partition pruning to index joins (#32007) #32093

mjonss · 2022-02-02T14:04:34Z

What problem does this PR solve?

Issue Number: close #32007

Problem Summary:
Explicit partition selection SELECT ... FROM pt PARTITION(p) ... was not considered when buildTableReaderForIndexJoin().

What is changed and how it works?

Use the PartitionNames (if set) and do skip looking up contents in non-mentioned partitions.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fixed explicit partition selection in subselects.

ti-chi-bot · 2022-02-02T14:04:35Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

djshow832
tiancaiamao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

mjonss · 2022-02-02T16:18:19Z

/run-unit-test

hawkingrei · 2022-02-03T10:13:22Z

/run-unit-test

hawkingrei · 2022-02-03T15:46:10Z

/run-unit-test

sre-bot · 2022-02-06T16:20:19Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/91102dce63a0b1e91f29e23bd3dcc0d5e14f6d50

djshow832

I'm curious: does only IndexJoin have this problem?

mjonss · 2022-02-10T10:08:23Z

I'm curious: does only IndexJoin have this problem?

Great question! Not sure, but I just tried with Hash join and Merge join for the same query, and it does give the correct result.

select /*+ HASH_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;
select /*+ MERGE_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;

Notice that the EXPLAIN of the original query says it only uses partition:p0 which is wrong (i.e. the explain code uses the given partition (p0) but the index join does not implement it.

@djshow832 Which other types of joins needs to be tested?

djshow832 · 2022-02-11T02:53:32Z

executor/builder.go

+			if _, ok := explicitPartitionSelection[pid]; !ok {
+				continue
+			}


I think the partitions are already pruned by partitionProcessor.convertToIntSlice() indirectly through partitionPruning() in this code block.
Thus, I think the code branches where partitionPruning() is not called may have the problem.

djshow832 · 2022-02-11T02:57:17Z

I'm curious: does only IndexJoin have this problem?

Great question! Not sure, but I just tried with Hash join and Merge join for the same query, and it does give the correct result.
select /*+ HASH_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;
select /*+ MERGE_JOIN(t3) */ * from t3 where t3.a <> ALL (select t1.a from t1 partition (p0)) order by t3.a;
Notice that the EXPLAIN of the original query says it only uses partition:p0 which is wrong (i.e. the explain code uses the given partition (p0) but the index join does not implement it.

@djshow832 Which other types of joins needs to be tested?

Not only joins but also other operators, which are hard to be tested fully. So I think the better way is to go through the code to check why buildTableReaderForIndexJoin missed it and how do other operators check it.

tiancaiamao

I'm afraid this is not the only place that missing the partition selection.
buildIndexReaderForIndexJoin / buildIndexLookupReaderForIndexJoin may have similiar issues...

tiancaiamao · 2022-02-11T07:27:40Z

I think the better way is to go through the code to check why buildTableReaderForIndexJoin missed it and how do other operators check it.

The code is too complex, that's the real root cause for this kind of bug.
As you can see, the code sketch is ..

if v.IsCommonHandle() {
    if len(lookUpContents) > 0 && keyColumnsIncludeAllPartitionColumns(lookUpContents[0].keyCols, pe) {
        // buggy branch
    } else {
        partitionPruning()
    }
} else {
   if len(lookUpContents) > 0 && keyColumnsIncludeAllPartitionColumns(lookUpContents[0].keyCols, pe) {
        // buggy branch
   } else {
        partitionPruning()
   }
}

djshow832 · 2022-03-08T02:25:54Z

/merge

ti-chi-bot · 2022-03-08T02:26:00Z

This pull request has been accepted and is ready to merge.

Commit hash: b7f8943

mjonss · 2022-03-08T10:20:08Z

/run-unit-test

mjonss · 2022-03-08T16:21:08Z

/run-unit-test