executor: calibrate resource support tpch10 #47095

guo-shaoge · 2023-09-19T13:43:13Z

What problem does this PR solve?

Issue Number: close #47094

Problem Summary: part of tiflash resource control

What is changed and how it works?

static calibrate: use tpch10 with 10 concurrency as workload. And run for 20seconds on local machine(20cores)

MySQL [test]> calibrate resource workload TPCH_10;
+--------+
| QUOTA  |
+--------+
| 308306 |
+--------+
1 row in set (0.018 sec)

dynamic calibrate: just like the original logic.
1. RU_PER_SEC = RU/(cpu_usage/total_logical_core)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: guo-shaoge <shaoge1994@163.com>

codecov · 2023-09-19T13:59:37Z

Codecov Report

Merging #47095 (7958a49) into master (9626556) will decrease coverage by 0.4223%.
Report is 27 commits behind head on master.
The diff coverage is 59.8130%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #47095        +/-   ##
================================================
- Coverage   73.0374%   72.6152%   -0.4223%     
================================================
  Files          1338       1361        +23     
  Lines        399306     408150      +8844     
================================================
+ Hits         291643     296379      +4736     
- Misses        88826      93042      +4216     
+ Partials      18837      18729       -108

Flag	Coverage Δ
integration	`32.3230% <1.9047%> (?)`
unit	`73.0563% <59.8130%> (+0.0188%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9913% <ø> (ø)`
parser	`84.9473% <0.0000%> (-0.0148%)`	⬇️
br	`48.5054% <ø> (-4.5966%)`	⬇️

Signed-off-by: guo-shaoge <shaoge1994@163.com>

CabinfeverB · 2023-09-20T08:03:32Z

executor/internal/calibrateresource/calibrate_resource.go

+	// read bytes: 401799161689.0 / 20 / 20 = 1004497904.22
+	const cpuTimePerCPUPerSec float64 = 263.74
+	const readBytesPerCPUPerSec float64 = 1004497904.22
+	ruPerCPU := float64(ruCfg.CPUMsCost)*cpuTimePerCPUPerSec + float64(ruCfg.ReadBytesCost)*readBytesPerCPUPerSec


An additional question: Is the cost factor used by Tiflash to calculate RU obtained from PD?

No, the number is by checking tiflash log(105494.666484 and 401799161689.0)

executor/internal/calibrateresource/calibrate_resource.go

Signed-off-by: guo-shaoge <shaoge1994@163.com>

CabinfeverB · 2023-09-20T10:25:43Z

/test unit-test

tiprow · 2023-09-20T10:26:18Z

@CabinfeverB: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test tiprow_fast_test

Use /test all to run all jobs.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge · 2023-09-20T12:39:09Z

/run-unit-test

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot · 2023-09-20T16:04:06Z

[LGTM Timeline notifier]

Timeline:

2023-09-20 08:05:44.327661423 +0000 UTC m=+677510.295249474: ☑️ agreed by CabinfeverB.
2023-09-20 16:04:05.002442887 +0000 UTC m=+706210.970030963: ☑️ agreed by nolouch.

HuSharp

rest LGTM!

Do we need to update dashboard and doc-cn?
https://github.com/pingcap/tidb-dashboard/blob/53563e713a2acce53a480e959f5d5c221caac993/ui/packages/tidb-dashboard-lib/src/apps/ResourceManager/translations/zh.yaml#L27C1-L30
please paste a SQL statement in the pr description :)

HuSharp · 2023-09-21T04:46:01Z

executor/internal/calibrateresource/calibrate_resource.go

+	return nil
+}
+
+func (e *Executor) getTiDBQuota(ctx context.Context, exec sqlexec.RestrictedSQLExecutor, startTs, endTs time.Time) (float64, error) {


Seems like it includes TiKV Quota, does it make sense to be called getTiDBQuota?

maybe getTiDBAndTiKVQuota ?

HuSharp · 2023-09-21T04:49:13Z

executor/internal/calibrateresource/calibrate_resource.go

-	quota := sum / float64(upperBound-lowerBound)
-	req.AppendUint64(0, uint64(quota))
-	return nil
+	return sum / float64(upperBound-lowerBound), nil


Could upperBound-lowerBound be zero?

It cannot be zero.
because len(quotas) >= 2. So len(quotas) must be greater than round(0.1*len(quotas))

easonn7 · 2023-09-21T07:13:43Z

/approve

There is no sysvar change.

ti-chi-bot · 2023-09-21T07:21:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CabinfeverB, easonn7, nolouch, wshwsh12, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XuHuaiyu,wshwsh12]
~~executor/OWNERS~~ [XuHuaiyu]
~~parser/OWNERS~~ [easonn7]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

guo-shaoge · 2023-09-21T08:12:15Z

/retest

guo-shaoge · 2023-09-21T09:13:13Z

/retest

guo-shaoge · 2023-09-21T09:39:37Z

/hold

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge · 2023-09-21T16:05:18Z

/retest

wuhuizuo · 2023-09-22T03:25:48Z

/retest

guo-shaoge · 2023-09-22T05:11:18Z

/retest

guo-shaoge · 2023-09-22T05:21:10Z

/retest

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge · 2023-09-22T07:02:40Z

/rebuild

guo-shaoge · 2023-09-22T07:03:30Z

/retest

guo-shaoge added 11 commits August 13, 2023 17:54

tmp save

8fc8fa2

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix empty bug

4a3323d

Signed-off-by: guo-shaoge <shaoge1994@163.com>

dyn done

6911e06

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fmt

d589b3e

Signed-off-by: guo-shaoge <shaoge1994@163.com>

static calibrate

015dc2e

Signed-off-by: guo-shaoge <shaoge1994@163.com>

add case; tpch10

26bbf2d

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'master' of github.com:pingcap/tidb into calibrate

6a53d0f

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix case

d4c9ceb

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix case

2a6f4e4

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix fmt && del log

ace33f5

Signed-off-by: guo-shaoge <shaoge1994@163.com>

enable resource control for batch cop

bb8b5cd

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot bot added do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-tests-checked labels Sep 19, 2023

revert tiflash semi join related fix

ada8cdd

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge added 3 commits September 19, 2023 22:03

revert rg name req code

505a539

Signed-off-by: guo-shaoge <shaoge1994@163.com>

bazel prepare

9619e49

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix lint

6c9c338

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge requested a review from HuSharp September 20, 2023 02:06

guo-shaoge added 2 commits September 20, 2023 10:37

refine func

585f99e

Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix tab

adec9eb

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge requested a review from nolouch September 20, 2023 06:51

del TPCH10 baseCost struct

216cdd7

Signed-off-by: guo-shaoge <shaoge1994@163.com>

CabinfeverB approved these changes Sep 20, 2023

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 20, 2023

fix comment

f86df62

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge added 2 commits September 20, 2023 19:29

Merge branch 'master' of github.com:pingcap/tidb into calibrate

fae0271

disable resource control by empty rgname

15e4a7d

Signed-off-by: guo-shaoge <shaoge1994@163.com>

bazel

09b47a4

Signed-off-by: guo-shaoge <shaoge1994@163.com>

nolouch approved these changes Sep 20, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 20, 2023

HuSharp reviewed Sep 21, 2023

View reviewed changes

wshwsh12 approved these changes Sep 21, 2023

View reviewed changes

XuHuaiyu approved these changes Sep 21, 2023

View reviewed changes

ti-chi-bot bot added the approved label Sep 21, 2023

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 21, 2023

guo-shaoge removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 21, 2023

del log

9d069a2

Signed-off-by: guo-shaoge <shaoge1994@163.com>

bazel

7958a49

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot bot merged commit 7a116f3 into pingcap:master Sep 22, 2023
12 of 16 checks passed

Oreoxmt mentioned this pull request Oct 16, 2023

keywords: add TPCH_10 and update the check-keywords.sh script pingcap/docs#15069

Merged

19 tasks

ti-chi-bot mentioned this pull request Oct 16, 2023

keywords: add TPCH_10 and update the check-keywords.sh script (#15069) pingcap/docs#15071

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: calibrate resource support tpch10 #47095

executor: calibrate resource support tpch10 #47095

guo-shaoge commented Sep 19, 2023 •

edited

Loading

codecov bot commented Sep 19, 2023 •

edited

Loading

CabinfeverB Sep 20, 2023

guo-shaoge Sep 20, 2023 •

edited

Loading

CabinfeverB commented Sep 20, 2023

tiprow bot commented Sep 20, 2023

guo-shaoge commented Sep 20, 2023

ti-chi-bot bot commented Sep 20, 2023

HuSharp left a comment •

edited

Loading

HuSharp Sep 21, 2023

guo-shaoge Sep 21, 2023

HuSharp Sep 21, 2023

guo-shaoge Sep 21, 2023

easonn7 commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

wuhuizuo commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

executor: calibrate resource support tpch10 #47095

executor: calibrate resource support tpch10 #47095

Conversation

guo-shaoge commented Sep 19, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

codecov bot commented Sep 19, 2023 • edited Loading

Codecov Report

CabinfeverB Sep 20, 2023

Choose a reason for hiding this comment

guo-shaoge Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

CabinfeverB commented Sep 20, 2023

tiprow bot commented Sep 20, 2023

guo-shaoge commented Sep 20, 2023

ti-chi-bot bot commented Sep 20, 2023

[LGTM Timeline notifier]

HuSharp left a comment • edited Loading

Choose a reason for hiding this comment

HuSharp Sep 21, 2023

Choose a reason for hiding this comment

guo-shaoge Sep 21, 2023

Choose a reason for hiding this comment

HuSharp Sep 21, 2023

Choose a reason for hiding this comment

guo-shaoge Sep 21, 2023

Choose a reason for hiding this comment

easonn7 commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

guo-shaoge commented Sep 21, 2023

wuhuizuo commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 22, 2023

guo-shaoge commented Sep 19, 2023 •

edited

Loading

codecov bot commented Sep 19, 2023 •

edited

Loading

guo-shaoge Sep 20, 2023 •

edited

Loading

HuSharp left a comment •

edited

Loading