Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry pick remaining commits from raftstore-proxy-6.1 to raftstore-proxy #72

Merged
merged 552 commits into from
May 30, 2022

Conversation

CalvinNeo
Copy link
Member

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

yuqi1129 and others added 30 commits December 15, 2021 14:08
…1269)

* ctl: should check column name validation when scan data using ctl

close tikv#11268
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
…onsistency (tikv#10977)

* Add mutation checking logic

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix usages of mutation

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto and renaming

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a simple test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix incorrect assertion

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Support check existance during pessimistic lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add assertion level support

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add more test cases

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test for check_existence of pessimistic_lock request

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* ref tikv#11542

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a metric for counting different way of checking assertion, for reference about the performance impact

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* close tikv#11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

* close tikv#11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…kv#11216)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* fix enable_ttl

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* simplify test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: qupeng <qupeng@pingcap.com>
* close tikv#11642

Signed-off-by: qupeng <qupeng@pingcap.com>

* a more comment

Signed-off-by: qupeng <qupeng@pingcap.com>
* ctl: split into multiple modules

ref tikv#10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* refactor if let to match

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* close tikv#11676

Signed-off-by: devillove084 <786537003@qq.com>
* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of
  the (i|u|f)(8|16|32|64) types in standard.

ref tikv#11678

```release-note
NONE
```

Signed-off-by: Harold Dost <harolddost@gmail.com>
Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
close tikv#11690

Signed-off-by: mornyx <mornyx.z@gmail.com>
* file_system: support collect the read flow by proc

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix clippy

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Delete unused content & Adjust copyright text

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add bench_fetch_io_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Integrate thread_io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* record bytes in request

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix logical bug

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Update test write_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Delete unused & Rename func

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add test for read & Update test for write

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add buffered thread io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Replace DashMap with ThreadLocal

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Rename variables and function & Remove unused call

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Change fetch logic. close tikv#10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add AtomicIOBytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Split ThreadIOSentinel

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* add THREAD_IO_TOTAL for metrics bytes fetcher

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Adjust code style & fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* improve write_bytes test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…ocks (tikv#11591)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>
* add datasink trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* RAII removes on_reporter_closing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

ref tikv#11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* datasink -> data_sink

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* clippy: Fixing up clippy errors.

* Fix crate references
* Fix anonymous lifetimes references.

ref tikv#4301
Signed-off-by: Harold Dost <harolddost@gmail.com>

* Test Revert of Iterations

ref tikv#4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* clippy: Enable check for rust-2018-idioms

ref tikv#4301

Signed-off-by: Harold Dost <harolddost@gmail.com>
ref tikv#11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: linning <linningde25@gmail.com>
* Use generic for api version to reduce runtime branching

ref tikv#10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* add test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fmt

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* rename vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* improve vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* sst_importer: check api version when importing

ref tikv#10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix bugs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* fix bug of Iterator::{scan,scan_cf}

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* remove redundant log

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Show the entire key

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix lint issue

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* resolve conflict

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
* br: sperate io and scan threads

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: fix a dummy bug

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: add config of io-threads

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: ref tikv#11350: some minior change

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: better tuning concurrency, and added a metric

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some slow log and metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: fix build"

This reverts commit 74537be.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: address comments"

This reverts commit 77d7575.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: always set IO type to Export

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup:make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>
ref tikv#11662

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* Update kvproto

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Return api-version for br

ref tikv#10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Reformat code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Update components/external_storage/export/src/export.rs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* format code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
* limit thread_io target os

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* limit thread_io target os. typo. close tikv#11698

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
Close tikv#10688

Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (danburkert/fs2-rs#42).

* bypass file allocate implementation in fs2

Signed-off-by: tabokie <xy.tao@outlook.com>
Minimum legend isn't very useful, and it occupied valuable visual real estate.

Remove some min legends from Grafana metrics, except for the following:
- Level 0 chance
- Number files at each level
- File Count
- Entry Count
- Allocator Stats
- Encrypted files
- Encryption meta files size

Ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>
…ation (tikv#11708)

* not return rows when there is no input for simple aggregation

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* close tikv#11735, and address comments

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
…ne by default (tikv#11699)

* backup: clamp auto tune values

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* ref tikv#11000: change the default backup threads.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run tasks in background

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run rustfmt

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: set remain threads to 20% of vcpu

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>
* s3: add request metrics for s3 storage

Signed-off-by: 3pointer <luancheng@pingcap.com>

* s3: add grafana json

Signed-off-by: 3pointer <luancheng@pingcap.com>

* br: add config for s3 multi part upload

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* hidden the new config

Signed-off-by: 3pointer <luancheng@pingcap.com>

* only hidden config in config-template.toml

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* close tikv#11727 && format

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref tikv#11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
CalvinNeo and others added 23 commits April 8, 2022 17:46
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
…kv#12315)

close tikv#12278, ref tikv#12309

Fix test_delegate_subscribe_unsubscribe flaking test. 
Use the corresponding id, otherwise there will be failures due to concurrent runs of the test. (We will assign downstream ids concurrently).

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>
Co-authored-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
f
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: Wish <breezewish@outlook.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
…g on ARM (pingcap#61)

Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
close tikv#11379, close tikv#11598, close tikv#12107

when memory usage is high water, add flow control on the append log entries.
disable log entry cache evict functions.

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
* raftstore: Introduce force leader state (tikv#11932)

close tikv#6107, ref tikv#10483

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* raftstore: Wait ticks for hibernated peer when doing force leader (tikv#12364)

ref tikv#10483

Force leader is rejected on a peer who is already a leader. For the hibernated leader,
it doesn't step down to follower when quorum is missing due to not ticking election. 
So wait ticks in force leader process for hibernated peers to make sure election ticking
is performed.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Make unsafe recovery wait apply cover snapshot apply cases ref tikv#10483 (tikv#12308)

ref tikv#10483

Signed-off-by: v01dstar <yang.zhang@pingcap.com>

* raftstore: Execute recovery plan via raft (tikv#12022)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Yang Zhang <yang.zhang@pingcap.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
@CLAassistant
Copy link

CLAassistant commented May 30, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 8 committers have signed the CLA.

✅ CalvinNeo
✅ mornyx
✅ breezewish
✅ purelind
✅ tier-cap
❌ ti-srebot
❌ tabokie
❌ Connor1996
You have signed the CLA already but the status is still pending? Let us recheck it.

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
@CalvinNeo CalvinNeo marked this pull request as ready for review May 30, 2022 06:57
@CalvinNeo CalvinNeo merged commit fbd13ee into pingcap:raftstore-proxy May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.