Adjust the store limit according to the cluster state dynamically #1902

shafreeck · 2019-11-06T06:57:18Z

~~Notice: This PR only could be merged after #1875 and #1887~~

What problem does this PR solve?

Adjust the store limit according to the cluster state dynamically

One of the main job of PD is to keep the data balanced on all TiKV nodes. The scheduler in PD moves regions between TiKVs when the unbalance is detected. In the case of adding or removing a TiKV node, the unbalance occurs, and the scheduler tries to move a lot amount of data. To avoid overloading a TiKV node, PD limits the speed to move regions for each TiKV node, the strategy is called store-limit.

The store-limit is a fixed value which is 15 regions per minute by default. This value is conservative, it prevents the TiKV from overloading in almost all cases. However, the value is too low for an idle cluster which may have no or little load. It takes too much time to balance the data when adding or removing a node. This PR tries to solve this problem by introducing the measurement of cluster state and adjust the store-limit dynamically according to the cluster state.

What is changed and how it works?

Leverage PR Calculate the state of the cluster using the CPU usages reported by TiKV #1875, we have basic statistics for the cluster state which define the cluster as idle if there are no keys or bytes accessed or the CPU usage is low.
The state of the cluster is calculated when receiving a store heartbeat, and the store limit is updated according to the latest state.
A configuration called StoreLimitScene is added for configuring the store-limit for different states, the valid states could be: idle, low, normal, high. The configuration can be updated by the HTTP API or pd-ctl.
Add a new subcommand for store: store limit-scene [<scene> <rate>]

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Code changes

Has HTTP API interfaces change

Related changes

Need to update the documentation
Need to be included in the release notes

shafreeck · 2019-11-06T09:45:35Z

/run-all-tests

codecov-io · 2019-11-21T07:40:00Z

Codecov Report

Merging #1902 into master will decrease coverage by 0.04%.
The diff coverage is 69.79%.

@@            Coverage Diff             @@
##           master    #1902      +/-   ##
==========================================
- Coverage   76.79%   76.75%   -0.05%     
==========================================
  Files         186      188       +2     
  Lines       19049    19162     +113     
==========================================
+ Hits        14628    14707      +79     
- Misses       3305     3335      +30     
- Partials     1116     1120       +4

Impacted Files	Coverage Δ
server/server.go	`82.98% <ø> (+0.38%)`	⬆️
server/cluster/cluster_stat.go	`79.56% <100%> (ø)`
server/config/config.go	`86.17% <100%> (+0.46%)`	⬆️
server/schedule/store_limit.go	`86.95% <100%> (ø)`	⬆️
server/schedule/store_limit_scenes.go	`100% <100%> (ø)`
server/cluster/metrics.go	`100% <100%> (ø)`	⬆️
server/handler.go	`50.42% <100%> (+0.21%)`	⬆️
server/api/router.go	`99.23% <100%> (+0.01%)`	⬆️
server/cluster/store_limiter.go	`53.65% <53.65%> (ø)`
server/cluster/cluster.go	`81.16% <60%> (-0.13%)`	⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a607642...8c251f7. Read the comment docs.

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

state Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

The store limit is wrapped by a struct called StoreLimit, it exports the useful api of the ratelimit.Bucket. It records the limit's setting mode which can be "auto" or "manual". An "manual" set value can overwrite an "auto set value, otherwise it is forbidden. Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

server/cluster/store_limiter.go

server/schedule/store_limit_scenes.go

server/cluster/metrics.go

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

shafreeck · 2019-12-26T07:24:34Z

@rleungx @lhy1024 PTAL

lhy1024

LGTM. Do we need to add a simple comparison test result under this PR?

shafreeck · 2019-12-26T07:39:06Z

LGTM. Do we need to add a simple comparison test result under this PR?

Do you have any ideas about the test?

rleungx

LGTM

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

shafreeck · 2019-12-27T08:18:57Z

/run-all-tests

sre-bot · 2019-12-27T08:19:38Z

/run-all-tests

sre-bot · 2019-12-27T08:21:45Z

@shafreeck merge failed.

shafreeck · 2019-12-27T08:51:47Z

/run-all-tests

shafreeck · 2019-12-27T12:05:11Z

/run-all-tests

nolouch · 2019-12-28T08:30:13Z

/merge

sre-bot · 2019-12-28T08:32:44Z

/run-all-tests

sre-bot · 2019-12-28T08:34:48Z

@shafreeck merge failed.

nolouch · 2019-12-30T06:29:00Z

/merge

sre-bot · 2019-12-30T06:29:09Z

/run-all-tests

sre-bot · 2019-12-30T06:32:41Z

@shafreeck merge failed.

shafreeck · 2019-12-30T07:40:33Z

/run-all-tests

shafreeck added type/enhancement The issue or PR belongs to an enhancement. DNM do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Nov 6, 2019

shafreeck self-assigned this Nov 6, 2019

shafreeck requested review from disksing, rleungx and nolouch November 6, 2019 06:57

shafreeck changed the title ~~WIP: Adjust the store limit according to the cluster state dynamically~~ Adjust the store limit according to the cluster state dynamically Nov 7, 2019

shafreeck removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2019

shafreeck force-pushed the schedule-in-cases branch 4 times, most recently from 6e7f819 to a776c31 Compare November 8, 2019 12:49

shafreeck force-pushed the schedule-in-cases branch from a776c31 to 5afb7ba Compare November 20, 2019 08:11

shafreeck force-pushed the schedule-in-cases branch from 6f956f8 to 5ae3c34 Compare November 21, 2019 08:10

shafreeck removed the DNM label Nov 22, 2019

shafreeck force-pushed the schedule-in-cases branch 2 times, most recently from 55d1677 to c767ec4 Compare December 5, 2019 12:09

shafreeck added 10 commits December 11, 2019 17:53

cacluate the cluster state

a9e711e

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

add unittest for keys and bytes statistics

038067a

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: support to exclude store IDs

fcece83

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: fix the potential panic caused by a 0 interval

50e90ac

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: add bytes and keys as the statistics to return the cluster

68791c4

state Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: fix typo

4476173

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: use named results

5836f9f

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: use statistics's median filter to calculate cpu usage

692b69d

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

feature: add the store limit mode definition

f2a7fa5

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

rleungx reviewed Dec 20, 2019

View reviewed changes

server/cluster/store_limiter.go Show resolved Hide resolved

server/schedule/store_limit_scenes.go Show resolved Hide resolved

server/cluster/metrics.go Outdated Show resolved Hide resolved

shafreeck added 6 commits December 26, 2019 14:49

cluster: refine codes

61e6f81

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: remove unused code

e9b254f

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

grpc: remove empty line

bf68390

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: refine the for loops

5024753

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: refine the comment

52fecb5

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

cluster: remove empty line

9dc8cdf

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

lhy1024 approved these changes Dec 26, 2019

View reviewed changes

rleungx approved these changes Dec 27, 2019

View reviewed changes

shafreeck added 2 commits December 27, 2019 16:02

Merge branch 'master' into schedule-in-cases

d678c18

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

server: format codes

f9e76d7

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

shafreeck added the status/can-merge Indicates a PR has been approved by a committer. label Dec 27, 2019

Merge branch 'master' into schedule-in-cases

9fd4826

Merge branch 'master' into schedule-in-cases

8c251f7

shafreeck merged commit 73e4f0d into tikv:master Dec 30, 2019

Luffbee mentioned this pull request Jan 14, 2020

*: remove unnecessary executable permissions #2095

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust the store limit according to the cluster state dynamically #1902

Adjust the store limit according to the cluster state dynamically #1902

shafreeck commented Nov 6, 2019 •

edited

Loading

shafreeck commented Nov 6, 2019

codecov-io commented Nov 21, 2019 •

edited

Loading

shafreeck commented Dec 26, 2019

lhy1024 left a comment

shafreeck commented Dec 26, 2019

rleungx left a comment

shafreeck commented Dec 27, 2019

sre-bot commented Dec 27, 2019

sre-bot commented Dec 27, 2019

shafreeck commented Dec 27, 2019

shafreeck commented Dec 27, 2019

nolouch commented Dec 28, 2019

sre-bot commented Dec 28, 2019

sre-bot commented Dec 28, 2019

nolouch commented Dec 30, 2019

sre-bot commented Dec 30, 2019

sre-bot commented Dec 30, 2019

shafreeck commented Dec 30, 2019

Adjust the store limit according to the cluster state dynamically #1902

Adjust the store limit according to the cluster state dynamically #1902

Conversation

shafreeck commented Nov 6, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

shafreeck commented Nov 6, 2019

codecov-io commented Nov 21, 2019 • edited Loading

Codecov Report

shafreeck commented Dec 26, 2019

lhy1024 left a comment

Choose a reason for hiding this comment

shafreeck commented Dec 26, 2019

rleungx left a comment

Choose a reason for hiding this comment

shafreeck commented Dec 27, 2019

sre-bot commented Dec 27, 2019

sre-bot commented Dec 27, 2019

shafreeck commented Dec 27, 2019

shafreeck commented Dec 27, 2019

nolouch commented Dec 28, 2019

sre-bot commented Dec 28, 2019

sre-bot commented Dec 28, 2019

nolouch commented Dec 30, 2019

sre-bot commented Dec 30, 2019

sre-bot commented Dec 30, 2019

shafreeck commented Dec 30, 2019

shafreeck commented Nov 6, 2019 •

edited

Loading

codecov-io commented Nov 21, 2019 •

edited

Loading