Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge TiKV 6.0 into Proxy #55

Merged
merged 547 commits into from
May 18, 2022

Conversation

CalvinNeo
Copy link
Member

@CalvinNeo CalvinNeo commented Apr 20, 2022

What problem does this PR solve?

Issue Number: close pingcap/tiflash#4618

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

ethercflow and others added 30 commits December 11, 2021 02:01
…1213)

close tikv#11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
* reduce extra next/pre in row_scan
close tikv#11572
Signed-off-by: Jayice <1185430411@qq.com>

* reduce extra next/pre in row_scan
close tikv#11572
Signed-off-by: Jayice <1185430411@qq.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
close tikv#11593

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
* add snap-generator-pool-size for configurable snapshot generator thread pool

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* rename config macro

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
* add prompt for ctl
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* optimize duplicate codes:pd check
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* move pd check in get_pd_rpc_client
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>

* simplify the codes check host|data-dir
close tikv#11514
Signed-off-by: Jayice <1185430411@qq.com>
…1269)

* ctl: should check column name validation when scan data using ctl

close tikv#11268
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
…onsistency (tikv#10977)

* Add mutation checking logic

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix usages of mutation

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto and renaming

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a simple test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix incorrect assertion

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Support check existance during pessimistic lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add assertion level support

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add more test cases

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test for check_existence of pessimistic_lock request

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* ref tikv#11542

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a metric for counting different way of checking assertion, for reference about the performance impact

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* close tikv#11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

* close tikv#11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…kv#11216)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* fix enable_ttl

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* simplify test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: qupeng <qupeng@pingcap.com>
* close tikv#11642

Signed-off-by: qupeng <qupeng@pingcap.com>

* a more comment

Signed-off-by: qupeng <qupeng@pingcap.com>
* ctl: split into multiple modules

ref tikv#10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* refactor if let to match

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* close tikv#11676

Signed-off-by: devillove084 <786537003@qq.com>
* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of
  the (i|u|f)(8|16|32|64) types in standard.

ref tikv#11678

```release-note
NONE
```

Signed-off-by: Harold Dost <harolddost@gmail.com>
Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
close tikv#11690

Signed-off-by: mornyx <mornyx.z@gmail.com>
* file_system: support collect the read flow by proc

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix clippy

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Delete unused content & Adjust copyright text

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add bench_fetch_io_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Integrate thread_io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* record bytes in request

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix logical bug

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Update test write_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Delete unused & Rename func

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add test for read & Update test for write

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add buffered thread io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Replace DashMap with ThreadLocal

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Rename variables and function & Remove unused call

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Change fetch logic. close tikv#10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add AtomicIOBytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Split ThreadIOSentinel

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* add THREAD_IO_TOTAL for metrics bytes fetcher

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Adjust code style & fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* improve write_bytes test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
…ocks (tikv#11591)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>
* add datasink trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* RAII removes on_reporter_closing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

ref tikv#11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* datasink -> data_sink

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* clippy: Fixing up clippy errors.

* Fix crate references
* Fix anonymous lifetimes references.

ref tikv#4301
Signed-off-by: Harold Dost <harolddost@gmail.com>

* Test Revert of Iterations

ref tikv#4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* clippy: Enable check for rust-2018-idioms

ref tikv#4301

Signed-off-by: Harold Dost <harolddost@gmail.com>
ref tikv#11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: linning <linningde25@gmail.com>
* Use generic for api version to reduce runtime branching

ref tikv#10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* add test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fmt

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* rename vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* improve vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* sst_importer: check api version when importing

ref tikv#10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix bugs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* fix bug of Iterator::{scan,scan_cf}

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* remove redundant log

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Show the entire key

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix lint issue

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* resolve conflict

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
* br: sperate io and scan threads

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: fix a dummy bug

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: add config of io-threads

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: ref tikv#11350: some minior change

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: better tuning concurrency, and added a metric

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some slow log and metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: fix build"

This reverts commit 74537be.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: address comments"

This reverts commit 77d7575.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: always set IO type to Export

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup:make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>
ref tikv#11662

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
* Update kvproto

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Return api-version for br

ref tikv#10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Reformat code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Update components/external_storage/export/src/export.rs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* format code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
* limit thread_io target os

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* limit thread_io target os. typo. close tikv#11698

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
Close tikv#10688

Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (danburkert/fs2-rs#42).

* bypass file allocate implementation in fs2

Signed-off-by: tabokie <xy.tao@outlook.com>
CalvinNeo and others added 9 commits April 2, 2022 13:28
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
f
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
…kv#12315)

close tikv#12278, ref tikv#12309

Fix test_delegate_subscribe_unsubscribe flaking test. 
Use the corresponding id, otherwise there will be failures due to concurrent runs of the test. (We will assign downstream ids concurrently).

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>
Co-authored-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
@CLAassistant
Copy link

CLAassistant commented Apr 20, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 7 committers have signed the CLA.

✅ mornyx
✅ CalvinNeo
✅ purelind
✅ breezewish
✅ tier-cap
❌ ti-srebot
❌ tabokie
You have signed the CLA already but the status is still pending? Let us recheck it.

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
f
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
mornyx and others added 7 commits May 12, 2022 12:33
Signed-off-by: mornyx <mornyx.z@gmail.com>
…g on ARM (pingcap#61)

Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
close tikv#11379, close tikv#11598, close tikv#12107

when memory usage is high water, add flow control on the append log entries.
disable log entry cache evict functions.

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>
@CalvinNeo CalvinNeo merged commit 713b918 into pingcap:raftstore-proxy May 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update TiFlash Proxy to 6.0