schedule: operator limit for stores #1474

rleungx · 2019-03-22T08:33:44Z

What problem does this PR solve?

We can not control the operators' cost in a store.

What is changed and how it works?

This PR adds an overloaded filter for stores. If the operators' cost in a store exceeds StoreMaxScheduleCost, the store will be filtered.
Also if the total cost of current running operators exceeds MaxScheduleCost, the newly created operator will be not added into running operators.

Check List

Tests

Unit test

codecov-io · 2019-03-22T08:41:31Z

Codecov Report

Merging #1474 into master will decrease coverage by 0.55%.
The diff coverage is 55.14%.

@@            Coverage Diff             @@
##           master    #1474      +/-   ##
==========================================
- Coverage   68.07%   67.51%   -0.56%     
==========================================
  Files         159      159              
  Lines       15843    16134     +291     
==========================================
+ Hits        10785    10893     +108     
- Misses       4100     4269     +169     
- Partials      958      972      +14

Impacted Files	Coverage Δ
server/schedule/scheduler.go	`66.66% <ø> (ø)`	⬆️
server/schedulers/random_merge.go	`66.66% <100%> (+0.87%)`	⬆️
server/schedulers/balance_region.go	`82.6% <100%> (+0.15%)`	⬆️
server/schedule/namespace_checker.go	`75.38% <100%> (+0.38%)`	⬆️
server/cluster.go	`74.46% <100%> (ø)`	⬆️
server/schedule/filters.go	`77.83% <100%> (+1.26%)`	⬆️
server/schedulers/evict_leader.go	`83.33% <100%> (+0.4%)`	⬆️
server/api/router.go	`100% <100%> (ø)`	⬆️
server/option.go	`90.62% <100%> (+0.07%)`	⬆️
server/config.go	`82.18% <100%> (+0.1%)`	⬆️
... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1268d8b...9d340cf. Read the comment docs.

server/core/store_option.go

server/schedule/operator.go

conf/config.toml

server/schedule/operator_controller.go

nolouch

Rest LGTM

server/schedule/operator_controller.go

server/schedule/metrics.go

server/schedulers/label.go

rleungx · 2019-05-22T05:30:21Z

/rebuild

shafreeck

LGTM and can be merged for its correctness, however, the design is recommended to be discussed and improved later.

Signed-off-by: nolouch <nolouch@gmail.com>

Signed-off-by: Ryan Leung <rleungx@gmail.com>

rleungx · 2019-05-22T06:09:36Z

/rebuild

* tiny clean up code (#1526) Signed-off-by: Ryan Leung <rleungx@gmail.com> * config: enable use region storage by default (#1524) * config: default enable use region storage * schedulers: let hot region balance not affect by balance-region-scheduler-limit (#1522) Signed-off-by: nolouch <nolouch@gmail.com> * log: do not add stack to error log (#1532) * *: replace gofail with pingcap/failpoint (#1534) * *: replace gofail with pingcap/failpoint Signed-off-by: disksing <i@disksing.com> * fix typo Signed-off-by: disksing <i@disksing.com> * fix typo Signed-off-by: disksing <i@disksing.com> * Update server/cluster_test.go Co-Authored-By: Lonng <chris@lonng.org> * Update server/tso.go Co-Authored-By: Lonng <chris@lonng.org> * Update server/tso.go Co-Authored-By: Lonng <chris@lonng.org> * check enable failpoint result Signed-off-by: disksing <i@disksing.com> * Update server/cluster.go Co-Authored-By: Lonng <chris@lonng.org> * server: set timeout for MoveLeader (#1533) * server: set timeout for MoveLeader Signed-off-by: disksing <i@disksing.com> * client, server: add ScanRegions gRPC protocol support (#1535) * client, server: support ScanRegions gRPC protocol Signed-off-by: disksing <i@disksing.com> * schedule: actively push operator (#1536) * schedule: actively push operator Signed-off-by: nolouch <nolouch@gmail.com> * *: update some dead links (#1543) * update links Signed-off-by: Ryan Leung <rleungx@gmail.com> * Add windows build script (#1540) Signed-off-by: Ana Hobden <operator@hoverbear.org> * fix operator timeout metrics (#1541) Signed-off-by: Ryan Leung <rleungx@gmail.com> * *:Rollback config in store when kv.persist failed (#1476) * tests: independent region sync test (#1545) * tests: independent syncer region test Signed-off-by: nolouch <nolouch@gmail.com> * schedule: operator limit for stores (#1474) * add store limit for scheduling Signed-off-by: Ryan Leung <rleungx@gmail.com> * statistic: add the statistic of flow (#1548) * store_statistic: add the statistic of flow Signed-off-by: nolouch <nolouch@gmail.com>

rleungx force-pushed the limit-store branch 2 times, most recently from a7df917 to a1a1921 Compare March 22, 2019 10:13

rleungx added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 22, 2019

rleungx force-pushed the limit-store branch from a1a1921 to b6bfd73 Compare March 24, 2019 09:26

rleungx added DNM and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Mar 25, 2019

rleungx force-pushed the limit-store branch 2 times, most recently from 2a41999 to 2df2083 Compare April 2, 2019 09:41

disksing reviewed Apr 8, 2019

View reviewed changes

server/core/store_option.go Outdated Show resolved Hide resolved

server/schedule/operator.go Outdated Show resolved Hide resolved

server/schedule/operator.go Outdated Show resolved Hide resolved

server/schedule/operator.go Outdated Show resolved Hide resolved

rleungx force-pushed the limit-store branch 2 times, most recently from e93bf61 to aa79ff9 Compare April 30, 2019 05:58

rleungx force-pushed the limit-store branch 9 times, most recently from 7643be0 to 5713c77 Compare May 7, 2019 11:50

nolouch reviewed May 8, 2019

View reviewed changes

conf/config.toml Show resolved Hide resolved

server/schedule/operator_controller.go Show resolved Hide resolved

rleungx requested review from nolouch and disksing and removed request for nolouch May 10, 2019 07:09

nolouch approved these changes May 13, 2019

View reviewed changes

server/schedule/operator_controller.go Show resolved Hide resolved

nolouch removed the DNM label May 20, 2019

disksing reviewed May 21, 2019

View reviewed changes

server/schedule/metrics.go Show resolved Hide resolved

disksing reviewed May 21, 2019

View reviewed changes

server/schedulers/label.go Show resolved Hide resolved

shafreeck approved these changes May 22, 2019

View reviewed changes

nolouch and others added 21 commits May 22, 2019 14:02

server: fix cannot send error in heartbeat stream (tikv#1521)

c5dbd1a

Signed-off-by: nolouch <nolouch@gmail.com>

add store limit for scheduling

edd18d6

Signed-off-by: Ryan Leung <rleungx@gmail.com>

add overloadFilter

ded7b1c

Signed-off-by: Ryan Leung <rleungx@gmail.com>

tiny change

240c014

Signed-off-by: Ryan Leung <rleungx@gmail.com>

set MaxScheduleCost to 0 by default

f79e2e8

Signed-off-by: Ryan Leung <rleungx@gmail.com>

use ratelimit for store

ecfc77f

Signed-off-by: Ryan Leung <rleungx@gmail.com>

rename exceedStoreCost to exceedStoreLimit

b10fd6e

Signed-off-by: Ryan Leung <rleungx@gmail.com>

adjust configurations

4039917

Signed-off-by: Ryan Leung <rleungx@gmail.com>

add pd-ctl limit command

4bf13be

Signed-off-by: Ryan Leung <rleungx@gmail.com>

add metrics for limiter

0d9a595

Signed-off-by: Ryan Leung <rleungx@gmail.com>

tiny fix

0c08cab

Signed-off-by: Ryan Leung <rleungx@gmail.com>

add show command

ab77a91

Signed-off-by: Ryan Leung <rleungx@gmail.com>

tiny fix and clean up

14991bb

Signed-off-by: Ryan Leung <rleungx@gmail.com>

adjust TolerantSizeRatio automatically

8ad268c

Signed-off-by: Ryan Leung <rleungx@gmail.com>

change the meaning of bucket rate

480c637

Signed-off-by: Ryan Leung <rleungx@gmail.com>

remove explicitly setting compacity

8ccf0d9

Signed-off-by: Ryan Leung <rleungx@gmail.com>

improve the limit metrics

206b606

Signed-off-by: Ryan Leung <rleungx@gmail.com>

address comment

777ceff

Signed-off-by: Ryan Leung <rleungx@gmail.com>

address comments

b4adbb5

Signed-off-by: Ryan Leung <rleungx@gmail.com>

remove unnecessary comment

a04f6a2

Signed-off-by: Ryan Leung <rleungx@gmail.com>

remove overloadfilter from region_scatterer

9d340cf

Signed-off-by: Ryan Leung <rleungx@gmail.com>

rleungx force-pushed the limit-store branch from ec079d7 to 9d340cf Compare May 22, 2019 06:03

Merge branch 'master' into limit-store

4a92eb7

nolouch merged commit 0fe66a0 into tikv:master May 22, 2019

rleungx mentioned this pull request May 30, 2019

schedule: add light peer without considering the influence #1554

Merged

rleungx mentioned this pull request Mar 2, 2020

UCP: Support rate control for removing peer #2178

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schedule: operator limit for stores #1474

schedule: operator limit for stores #1474

rleungx commented Mar 22, 2019

codecov-io commented Mar 22, 2019 •

edited

Loading

nolouch left a comment

rleungx commented May 22, 2019

shafreeck left a comment

rleungx commented May 22, 2019

schedule: operator limit for stores #1474

schedule: operator limit for stores #1474

Conversation

rleungx commented Mar 22, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

codecov-io commented Mar 22, 2019 • edited Loading

Codecov Report

nolouch left a comment

Choose a reason for hiding this comment

rleungx commented May 22, 2019

shafreeck left a comment

Choose a reason for hiding this comment

rleungx commented May 22, 2019

codecov-io commented Mar 22, 2019 •

edited

Loading