copr: allow checksum request to be rescheduled (#9094)

[ti-srebot]

cherry-pick #9094 to release-4.0

---

What problem does this PR solve?

Issue Number: close pingcap/br#611

Problem Summary:

Running the checksum coprocessor, even with concurrency = 1, affects the cluster performance since it never yields control.

What is changed and how it works?

What's Changed:

Copied the "reschedule" mechanism from the "analyze" coprocessor to the checksum coprocessor.

Related changes

• Need to cherry-pick to the release branch
  • 3.0, 4.0

Check List

Tests

• Manual test (add detailed scripts or steps below)
  • Execute BR backup while a SELECT is running in background. The following shows the TiDB QPS measurement before and after applying this PR:
    
    • Green = (Negation of) number of keys processed by the "checksum" coprocessor, non-zero means a checksum is running
    • Yellow = QPS, higher means the background SELECT is less affected
    • Left 3 = before this PR was applied, QPS is dropped to 30% of original
    • Right 1 = after this PR is applied, QPS is almost unaffected.

Side effects

Release note

• Running checksum in BR and Lightning should have less influence on the cluster performance.

[ti-srebot]

/run-all-tests

[ti-srebot]

@kennytm you're already a collaborator in bot's repo.

[kennytm]

/test

https://internal.pingcap.net/idc-jenkins/blue/rest/organizations/jenkins/pipelines/tikv_ghpr_test/runs/36024/nodes/212/log/?start=0

[2020-11-24T09:56:29.341Z] test raftstore::test_region_heartbeat::test_server_pending_peers ... thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("[components/test_raftstore/src/cluster.rs:331]: can\'t get leader of region 1")', src/libcore/result.rs:1188:5
[2020-11-24T09:56:29.341Z] stack backtrace:
[2020-11-24T09:56:29.341Z]   14: core::result::Result<T,E>::unwrap
[2020-11-24T09:56:29.341Z]              at /rustc/0de96d37fbcc54978458c18f5067cd9817669bc8/src/libcore/result.rs:956
[2020-11-24T09:56:29.341Z]   15: test_raftstore::cluster::Cluster<T>::request
[2020-11-24T09:56:29.341Z]              at ./components/test_raftstore/src/cluster.rs:671
[2020-11-24T09:56:29.341Z]   16: test_raftstore::cluster::Cluster<T>::must_put_cf
[2020-11-24T09:56:29.341Z]              at ./components/test_raftstore/src/cluster.rs:816
[2020-11-24T09:56:29.341Z]   17: test_raftstore::cluster::Cluster<T>::must_put
[2020-11-24T09:56:29.341Z]              at ./components/test_raftstore/src/cluster.rs:812
[2020-11-24T09:56:29.341Z]   18: integrations::raftstore::test_region_heartbeat::test_pending_peers
[2020-11-24T09:56:29.341Z]              at tests/integrations/raftstore/test_region_heartbeat.rs:128
[2020-11-24T09:56:29.341Z]   19: integrations::raftstore::test_region_heartbeat::test_server_pending_peers
[2020-11-24T09:56:29.341Z]              at tests/integrations/raftstore/test_region_heartbeat.rs:157
[2020-11-24T09:56:29.341Z]   20: integrations::raftstore::test_region_heartbeat::test_server_pending_peers::{{closure}}
[2020-11-24T09:56:29.341Z]              at tests/integrations/raftstore/test_region_heartbeat.rs:155
[2020-11-24T09:56:29.342Z] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2020-11-24T09:56:29.342Z] FAILED

[zhouqiang-cl]

/run-all-tests

[Little-Wallace]

/run-all-tests

[sticnarf]

LGTM

[Little-Wallace]

LGTM

[ti-srebot]

@Little-Wallace, Thanks for your review. The bot only counts LGTMs from Reviewers and higher roles, but you're still welcome to leave your comments. See the corresponding SIG page for more information. Related SIG: coprocessor(slack).

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 9125

[ti-srebot]

/run-all-tests