store/tikv: always switch to a different peer when meets no-leader (#22449)

[ti-srebot]

cherry-pick #22449 to release-5.0
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/23595

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tidb.git pr/23595:release-5.0-2df2ca06d27c

---

Signed-off-by: youjiali1995 zlwgx1023@gmail.com

What problem does this PR solve?

Issue Number: close #22400
Problem Summary:

Joint consensus is enabled in master, which is possible to make a leader step down as a learner during a conf change. And hibernate region is also enabled by default in master, so after the leader step down, there can be a long time that there is no leader in the region until requests are sent to followers or hibernate timeout. But from logs, I can see TiDB keeps querying the learner instead of trying other followers, hence it keeps timeout until hibernate is woken up.

What is changed and how it works?

What's Changed:

One solution is always to try a different peer when meets no-leader. There is a small probability that the current peer who reports not-leader becomes a leader and TiDB has to retry once in this case.

Related changes

Check List

Tests

• Unit test

Release note

• No release note

[ti-srebot]

/run-all-tests

[ti-srebot]

@youjiali1995 you're already a collaborator in bot's repo.

[cfzjywxk]

LGTM

[ti-chi-bot]

@cfzjywxk: Please use /LGTM instead of LGTM when you want to approve the pull request by comment.
If you use the GitHub review feature, please approve the PR directly, the comment will not take effect in the GitHub review feature.
If you have any qustions please refer to lgtm command help or lgtm plugin design.

If you have approved this PR, please ignore this reply. This reply is being used as a temporary reply during the migration of the new bot and will be removed on April 1.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• MyonKeminta
• youjiali1995

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

[youjiali1995]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 09c1ef4

[lysu]

/run-common-test
/run-integration-common-test

[youjiali1995]

/run-unit-test