The hot cache cannot be cleared, when the interval is less than 60

[lhy1024]

Bug Report

What did you do?

stop bench

What did you expect to see?

hot cache is cleared

What did you see instead?

there are still many hot peers

What version of PD are you using (pd-server -V)?

v5.0.4

[lhy1024]

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it collects 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

1. the store is the same, then the old item is used directly
2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

[lhy1024]

Our current heartbeat process looks like this.

If the interval is less than the default interval for heartbeats (60 seconds), then we will put it into the cache temporarily and wait until it has made up 60 seconds before considering whether it is hot enough.

For a peer that has just been reported, if the region is also in the hot cache, then there are three cases.

1. the store is the same, then the old item is used directly
2. the store is different, but there is a move peer or transfer leader in the previous round, then the inheritable item is directly used as the old item
3. store is different, and there is no inheritable peer, then choose any peer as old item

The problem occurs in the third branch, if the old peer is used directly without clone, then the old item and the new item will be written at the same time. And when the new peer interval is less than 60 seconds, it means that it will be temporarily put into the cache.

If the old peer will be cooled down at this time, it will keep the peer in the hot cache for a long time and cannot be exited.

For store is different, and there is no inheritable peer, then choose any peer as old item, we call it as adopt, taking the child of a relative as own son or daughter.

Suppose there is a cluster with region heartbeat 5s.

At t1, when a peer in store1 is changed to cold, and other peers still are hot, it will adopt from store2 as own, it is expected.

	source	hot/cold	interval of sum
store1	adopt from store2 at t1	cold	60s
store2	inherit	hot	55s
store3	inherit	hot	55s

But other peers will be changed to cold, the problem occurs. A peer in store2 or store3 adopts from store1 at t2, but this child is from store2 at t1.

	source	hot/cold	interval of sum
store1	adopt from store2 at t1	cold	5s
store2	adopt from store1 at t2 (it is from store2 at t1!)	cold	60s
store3	adopt from store1 at t2 (it is from store2 at t1!)	cold	60s

So we should avoid to adopt from adopted item before it is confirmed whether it is hot.

[nolouch]

/assign @lhy1024

[mayjiang0203]

/severity major