store: the peer id mismatch is not handled

[cfzjywxk]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Consider such a case:

1. region_id = 0
   There's a region with initial peers "1, 2, 3", and later region configuration has changed, for example:

1, 2, 3                    conf change => 1 2 3(Removed) 4(Added)
1, 2, 3(Removed), 4(Added) conf change => 1 2 5(Added) 4(Removed)

2. The region cache is not refreshed in time, so peer information whose store_id is 3, region_id is 0 andpeer_id is 3 still exists.
3. AccessFollower is used by the replica selector to decide the target peer for example when stale read is used.
4. If the leader is transferred to peer 5 and stale read retry happens.

2. What did you expect to see? (Required)

The region cache is refreshed as expected.

3. What did you see instead (Required)

After RegionError is received by the kv-client, the bottom retry keeps happening and there's no chance to rerefresh to invalidate the region cache, thus there are quite a lot of retry and keep encountering peer is mismatch region errors.

The cause is possibly that:

1. The peer id mismatch is not processed, the replica selector would always retry.
2. The scope fixing PR make the leader only selection take effect.
3. Once the leaderOnly flag is set, all the checks are bypassed so the const maxReplicaAttempt = 10 dose not work either.

To fix this issue, it's needed to:

1. Process the peer id mismatch error properly and ensure the region cache is invalidated.
2. When leaderOnly flag is set, ensure the necessary limit takes effect for example the const maxReplicaAttempt = 10. Acually this part code is complex and need careful handling.

4. What is your TiDB version? (Required)

v6.5.3

[you06]

Fixes on client-go:

• 6.5 handle mismatch peer id region error tikv/client-go#888
• 7.1 handle mismatch peer id region error tikv/client-go#885 & Match error message correctly tikv/client-go#887
• master handle mismatch peer id region error tikv/client-go#935

[cfzjywxk]

Fixes on client-go:

• 6.5 handle mismatch peer id region error tikv/client-go#888
• 7.1 handle mismatch peer id region error tikv/client-go#885 & Match error message correctly tikv/client-go#887
• master handle mismatch peer id region error tikv/client-go#935

Close this issue as it's fixed.