feat(region_cache): sync leader store epoch when switchWorkLeaderToPeer

[Ryan-Git]

If leader transfers to a store that disconnected for a while and comes back now, request should not be blocked by store epoch.

[sticnarf]

I'm not confident enough about this change.

The epoch in Store can be updated concurrently. When calling switchWorkLeaderToPeer, the epoch may have been updated again due to failed requests. Then, it may be not reasonable to always synchronize storeEpochs[leaderIdx] to the latest epoch in Store.

[Ryan-Git]

Yes it may cause a few more concurrent requests send to disconnected KV.

But since raft think the target store is still leader(likely just transferred leadership to it) and the response shouldn't be delayed for too long, it should be rare.

The good part is it avoids randomized retries afterward. In normal deployments, the restarted store will be leader balanced quickly, which causes quite a few fake epoch not match errors.

I'm testing this anyway... will post some numbers for reference later.

[disksing]

maybe we can read + CAS to update epoch and make sure it always increases.

[Ryan-Git]

a few result.

workload: medium length transaction (0-500 statements per txn, uniform distribution)
chaos: force kill one kv and immediately restarts every hour on the hour

updated version starts from 16:00

kv:20002 is eliminated

total error(mostly statement timeout) not increased and no unexpected error after restart

[sticnarf]

LGTM