executor: add retriver for tidb_hot_regions_history #27375

IcePigZDB · 2021-08-19T07:31:47Z

Signed-off-by: IcePigZDB icepigzdb@gmail.com

Problem Summary: Create a new table TIDB_HOT_REGIONS_HISTORY in the INFORMATION_SCHEMAT database to retrieve history hotspot information stored by PD periodically.

What problem does this PR solve?

Proposal: pingcap/tidb#27487
Associated Issue: #25281

Associated PRs in TiDB:
PRs splited from oversize PR #27224 in merge order :

Associated PR in PD:
PRs splited from old oversize one: tikv/pd/pull/3989

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

select * from information_schema.tidb_hot_regions_history where update_time>='2019-10-10 10:10:10' and update_time<'2019-10-11 10:10:10'

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

ti-chi-bot · 2021-08-19T07:31:48Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

nolouch
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

…xtractor

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

…e key Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

…heck Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

…x results order overlap Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

Signed-off-by: IcePigZDB <icepigzdb@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com>

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

nolouch · 2021-11-16T05:57:57Z

PTAL @rleungx

JmPotato · 2021-11-16T09:17:44Z

/uncc JmPotato

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

rleungx · 2021-11-17T08:33:54Z

executor/memtable_reader.go

+				sctx.GetSessionVars().StmtCtx.AppendWarning(result.err)
+			}
+			if result.err != nil || len(result.messages.HistoryHotRegion) == 0 {
+				continue


move continue to line 886 and check len(result.messages.HistoryHotRegion) > 0?

rleungx · 2021-11-17T08:37:44Z

executor/memtable_reader.go

+) ([]types.Datum, error) {
+	_, startKey, _ := codec.DecodeBytes(hisHotRegion.StartKey, []byte{})
+	_, endKey, _ := codec.DecodeBytes(hisHotRegion.EndKey, []byte{})
+	region := &tikv.KeyLocation{StartKey: startKey, EndKey: endKey}


Is there possible the old table is dropped?

yes, we have discussed this case, dropped tables' hot region will be ignore in current implement.

_, startKey, _ := codec.DecodeBytes(hisHotRegion.StartKey, []byte{}) _, endKey, _ := codec.DecodeBytes(hisHotRegion.EndKey, []byte{}) region := &tikv.KeyLocation{StartKey: startKey, EndKey: endKey} hotRange, err := helper.NewRegionFrameRange(region) if err != nil { return nil, err } f := tikvHelper.FindTableIndexOfRegion(allSchemas, hotRange) // Ignore row without corresponding schema f. if f == nil { return nil, nil }

This case has been tested in memtable_reader_test.go/TestTiDBHotRegionsHistory line 1097-1099, code run as expected.

// table_id = 131, index_id = 1, index_value = 1, deleted schema {"2019-10-10 10:10:23", "UNKONW", "UNKONW", "131", "UNKONW", "1", "5", "5", "55555", "0", "1", "READ", "99", "99", "99", "99"}, {"2019-10-10 10:10:24", "UNKONW", "UNKONW", "131", "UNKONW", "1", "6", "6", "66666", "0", "0", "WRITE", "99", "99", "99", "99"},

rleungx · 2021-11-17T08:39:00Z

planner/core/memtable_predicate_extractor.go

@@ -541,6 +541,24 @@ func (helper extractHelper) convertToTime(t int64) time.Time {
 	return time.Unix(0, t)
 }

+func (helper extractHelper) extractRoleBoolSliceFromUint64Slice(uint64Slice []uint64) []bool {


The function name is too long.

change to convertToBoolSlice according to previous func convertToTime.

rleungx · 2021-11-17T08:40:05Z

planner/core/util.go

+	for _, k := range slice {
+		l = append(l, fmt.Sprintf(`%t`, k))
+	}
+	sort.Strings(l)


What is the purpose of sort?

In unnormal case, user may query is_leader in(0,1), sort can keep the order of result string.

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

Yisaer · 2021-11-19T08:30:53Z

/merge

ti-chi-bot · 2021-11-19T08:30:57Z

This pull request has been accepted and is ready to merge.

Commit hash: e4d113f

executor: add retriver for tidb_hot_regions_history

d8f5844

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 19, 2021

This was referenced Aug 19, 2021

infoschema: add tidb_hot_regions_history virtual table #27373

Merged

planner: add extractor for tidb_hot_regions_history #27374

Merged

github-actions bot added the sig/execution SIG execution label Aug 19, 2021

IcePigZDB mentioned this pull request Aug 19, 2021

infoschema,planner,execuor: support hot history region #27224

Closed

12 tasks

IcePigZDB added 5 commits August 19, 2021 21:19

executor: unified use of ms in update_time, and move IntSet init to e…

f87a112

…xtractor

executor: fix time formate bug

644b406

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: fix history hot regions memtable reader test time bug

9067e13

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: call DecodeBytes for region range key, and update test rang…

4724738

…e key Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: change UPDATE_TIME type to TypeTimestamp and add timezone c…

78a9484

…heck Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

IcePigZDB force-pushed the hishotregionsexecutor branch from 143c248 to 78a9484 Compare August 19, 2021 13:23

IcePigZDB added 4 commits August 22, 2021 10:59

executor: devide read and write hot types into two http request to fi…

ff0c630

…x results order overlap Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: add is_leader in request and update retriver test

faf8f52

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: remove debug test case

0330dc6

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: close httpServers after test down

42a5c61

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

IcePigZDB force-pushed the hishotregionsexecutor branch from 2201462 to 42a5c61 Compare August 22, 2021 12:43

IcePigZDB mentioned this pull request Aug 23, 2021

docs: add proposal for TiDB_HOT_REGIONS_HISTORY #27487

Open

executor: roles from intset to slice

c8e440d

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

This was referenced Aug 23, 2021

cluster: add hot_region_storage tikv/pd#4020

Merged

api: add interface for hot_region info reqeust tikv/pd#4021

Merged

config: add hot region history configurations tikv/pd#4019

Merged

IcePigZDB added 6 commits August 23, 2021 11:49

execuotr: use bool for IsLeader

034533d

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: use http.MethodGet

6ba28be

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: remove extraction unnecessary columns

8008798

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: review from @rleungx

95093e9

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

exectuor:change PD-Allow-follower-handle to false

cb2cbed

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

executor: add is_learner field

0490b49

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

IcePigZDB force-pushed the hishotregionsexecutor branch from 3a87459 to f5b478b Compare October 20, 2021 08:26

planner: keep role bool slice unique

c8d8c11

Signed-off-by: IcePigZDB <icepigzdb@gmail.com> Co-authored-by: ShuNing <nolouch@gmail.com>

IcePigZDB force-pushed the hishotregionsexecutor branch from f5b478b to c8d8c11 Compare October 20, 2021 08:32

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 20, 2021

IcePigZDB and others added 2 commits October 20, 2021 08:35

planner: update comment

4ddc37f

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

Merge branch 'master' into hishotregionsexecutor

485b76f

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 2, 2021

Merge branch 'master' into hishotregionsexecutor

61dcc1b

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 9, 2021

Merge branch 'master' into hishotregionsexecutor

57b2144

ti-chi-bot removed the request for review from JmPotato November 16, 2021 09:17

IcePigZDB and others added 2 commits November 17, 2021 13:53

Merge branch 'master' into hishotregionsexecutor

957fe33

fix conflict due to the remove of pdapi.clusterVersion

e05a248

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

rleungx reviewed Nov 17, 2021

View reviewed changes

address comment

e4d113f

Signed-off-by: IcePigZDB <icepigzdb@gmail.com>

rleungx approved these changes Nov 18, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 18, 2021

nolouch approved these changes Nov 18, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 18, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 19, 2021

Merge branch 'master' into hishotregionsexecutor

fa68994

ti-chi-bot merged commit 177be48 into pingcap:master Nov 19, 2021

This was referenced Dec 8, 2021

executor: use GetRegionsTableInfo to get region schema for tidb_hot_regions_history #30518

Open

executor: use helper.GetRegionsTableInfo to get region schema for tidb_hot_regions_history #30500

Merged

IcePigZDB mentioned this pull request Jan 15, 2022

Add virtual table TiDB_HOT_REGIONS_HISTORY #27772

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: add retriver for tidb_hot_regions_history #27375

executor: add retriver for tidb_hot_regions_history #27375

IcePigZDB commented Aug 19, 2021 •

edited

Loading

ti-chi-bot commented Aug 19, 2021 •

edited

Loading

nolouch commented Nov 16, 2021

JmPotato commented Nov 16, 2021

rleungx Nov 17, 2021

IcePigZDB Nov 17, 2021

rleungx Nov 17, 2021

IcePigZDB Nov 17, 2021

rleungx Nov 17, 2021

IcePigZDB Nov 17, 2021

rleungx Nov 17, 2021

IcePigZDB Nov 17, 2021

Yisaer commented Nov 19, 2021

ti-chi-bot commented Nov 19, 2021

executor: add retriver for tidb_hot_regions_history #27375

executor: add retriver for tidb_hot_regions_history #27375

Conversation

IcePigZDB commented Aug 19, 2021 • edited Loading

What problem does this PR solve?

Check List

Release note

ti-chi-bot commented Aug 19, 2021 • edited Loading

nolouch commented Nov 16, 2021

JmPotato commented Nov 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yisaer commented Nov 19, 2021

ti-chi-bot commented Nov 19, 2021

IcePigZDB commented Aug 19, 2021 •

edited

Loading

ti-chi-bot commented Aug 19, 2021 •

edited

Loading