executor: push down LIKE patterns for some infoschema readers #55844

tangenta · 2024-09-04T07:53:24Z

What problem does this PR solve?

Issue Number: ref #50305

Problem Summary:

Previously, we only support pushing down LIKE pattern for tidb_index_usage and columns.

What changed and how does it work?

Support pushing down LIKE for other infoschema tables.
Merge ListSchemas and ListTables into ListSchemasAndTables for setDataFromIndexUsage and setDataForColumns.
Refactor hugeMemTableRetriever according to previous point.
Refactor pushed down mechanism for upper() and lower(), now it is much flexible to support other functions.
Remove InfoSchemaTableNameExtractor.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

…table_name columns

tiprow · 2024-09-04T07:53:40Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-09-04T08:07:02Z

Codecov Report

Attention: Patch coverage is 93.24895% with 16 lines in your changes missing coverage. Please review.

Project coverage is 57.3350%. Comparing base (70a26f8) to head (b78dc97).
Report is 30 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #55844         +/-   ##
=================================================
- Coverage   72.9579%   57.3350%   -15.6230%     
=================================================
  Files          1611       1760        +149     
  Lines        447514     643427     +195913     
=================================================
+ Hits         326497     368909      +42412     
- Misses       100924     250987     +150063     
- Partials      20093      23531       +3438

Flag	Coverage Δ
integration	`38.5859% <88.6877%> (?)`
unit	`73.4657% <91.8552%> (+1.3985%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.1514% <ø> (+0.1947%)`	⬆️
parser	`∅ <ø> (∅)`
br	`52.8257% <ø> (+7.0371%)`	⬆️

joechenrh · 2024-09-05T07:13:44Z

pkg/executor/infoschema_reader.go

@@ -927,7 +927,12 @@ func (e *hugeMemTableRetriever) retrieve(ctx context.Context, sctx sessionctx.Co

 	if !e.initialized {
 		e.is = sessiontxn.GetTxnManager(sctx).GetTxnInfoSchema()
-		e.dbs = e.extractor.ListSchemas(e.is)
+		dbs, tbls, err := e.extractor.ListSchemasAndTables(ctx, e.is)


I think the original design is to reduce the memory usage as much as possible. I'm not sure whether this will be a potential problem.

Maybe we can add a streaming interface for SchemaTableInfos to solve this completely.. Let me revert this change for now.

pkg/planner/core/memtable_infoschema_extractor.go

pkg/planner/core/memtable_predicate_extractor.go

joechenrh · 2024-09-06T05:57:59Z

pkg/planner/core/memtable_predicate_extractor.go

+	}
+	switch scalar.FuncName.L {
+	case ast.Lower:
+		helper.pushedDownFuncs = make(map[string]func(string) string)


pushedDownFuncs is created each time setColumnPushedDownFn called?

I reduce the number of times to call setColumnPushedDownFn in this commit

joechenrh · 2024-09-23T07:09:53Z

/lgtm

ti-chi-bot · 2024-09-23T07:10:01Z

@joechenrh: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot · 2024-09-23T07:12:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CbcWestwolf, hawkingrei, joechenrh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [CbcWestwolf,hawkingrei]
~~pkg/planner/OWNERS~~ [hawkingrei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-09-23T07:12:21Z

[LGTM Timeline notifier]

Timeline:

2024-09-20 07:17:11.759911315 +0000 UTC m=+1204701.500335255: ☑️ agreed by CbcWestwolf.
2024-09-23 07:12:20.688309349 +0000 UTC m=+1463610.428733295: ☑️ agreed by hawkingrei.

tangenta · 2024-09-23T09:37:28Z

/retest

tiprow · 2024-09-23T09:37:50Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2024-09-23T11:19:49Z

/retest

tiprow · 2024-09-23T11:20:11Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…p#55844) ref pingcap#50305

executor: support like pattern for infoschema reader on schema_name, …

4c8a132

…table_name columns

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 4, 2024

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Sep 4, 2024

tangenta added 4 commits September 4, 2024 17:03

add integration tests

14c845c

fix linter

061791d

test: fix TestInfoSchemaTableExtract

f78262e

update integration test result

c564314

joechenrh reviewed Sep 5, 2024

View reviewed changes

CbcWestwolf self-requested a review September 6, 2024 01:47

CbcWestwolf reviewed Sep 6, 2024

View reviewed changes

pkg/planner/core/memtable_infoschema_extractor.go Outdated Show resolved Hide resolved

joechenrh reviewed Sep 6, 2024

View reviewed changes

pkg/planner/core/memtable_predicate_extractor.go Show resolved Hide resolved

joechenrh reviewed Sep 6, 2024

View reviewed changes

tangenta added 3 commits September 9, 2024 16:52

address comment

be4f3de

Merge remote-tracking branch 'upstream/master' into HEAD

3d92835

revert setDataForColumns

b78dc97

tangenta requested review from joechenrh and CbcWestwolf September 19, 2024 06:46

CbcWestwolf approved these changes Sep 20, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 20, 2024

hawkingrei approved these changes Sep 23, 2024

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 23, 2024

ti-chi-bot bot merged commit 8c132db into pingcap:master Sep 23, 2024
23 checks passed

winoros pushed a commit to winoros/tidb that referenced this pull request Sep 23, 2024

executor: push down LIKE patterns for some infoschema readers (pingca…

79c6958

…p#55844) ref pingcap#50305

This was referenced Sep 27, 2024

information_schema.tables return wrong result #56377

Closed

planner: make pattern match case-insensitive for some infoschema tables #56378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: push down LIKE patterns for some infoschema readers #55844

executor: push down LIKE patterns for some infoschema readers #55844

tangenta commented Sep 4, 2024 •

edited

Loading

tiprow bot commented Sep 4, 2024

codecov bot commented Sep 4, 2024 •

edited

Loading

joechenrh Sep 5, 2024

tangenta Sep 5, 2024

tangenta Sep 19, 2024

joechenrh Sep 6, 2024

tangenta Sep 19, 2024

joechenrh commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

tangenta commented Sep 23, 2024

tiprow bot commented Sep 23, 2024

tangenta commented Sep 23, 2024

tiprow bot commented Sep 23, 2024

executor: push down LIKE patterns for some infoschema readers #55844

executor: push down LIKE patterns for some infoschema readers #55844

Conversation

tangenta commented Sep 4, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Sep 4, 2024

codecov bot commented Sep 4, 2024 • edited Loading

Codecov Report

joechenrh Sep 5, 2024

Choose a reason for hiding this comment

tangenta Sep 5, 2024

Choose a reason for hiding this comment

tangenta Sep 19, 2024

Choose a reason for hiding this comment

joechenrh Sep 6, 2024

Choose a reason for hiding this comment

tangenta Sep 19, 2024

Choose a reason for hiding this comment

joechenrh commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

ti-chi-bot bot commented Sep 23, 2024

[LGTM Timeline notifier]

tangenta commented Sep 23, 2024

tiprow bot commented Sep 23, 2024

tangenta commented Sep 23, 2024

tiprow bot commented Sep 23, 2024

tangenta commented Sep 4, 2024 •

edited

Loading

codecov bot commented Sep 4, 2024 •

edited

Loading