reduce locking contention for region heartbeat handling

[hihihuhu]

Enhancement Task

we are running a tidb cluster to host about 90TB data on 1.8m regions, and seeing significant locking contention that causes 2 issues when there are massive region movement, i.e. when scales out tikv.

1. region heartbeat handling itself, which could lead to pd OOM because of pending heartbeat requests
2. the latency GetRegion requests increase significantly because of the locking, thus make the sql queries timeout.

dig into pd code, the locks has coarse scope and could be improved to increase the throughput. for example,

1. https://github.com/tikv/pd/blob/v6.3.0/server/cluster/cluster.go#L809 doesn't have to lock everything but could use a per region/per region bucket lock
2. https://github.com/tikv/pd/blob/v6.3.0/server/core/basic_cluster.go#L30, this locks 2 different data structures, could be separated.
3. or even better, BasicCluster could only lock thing based on its business needs, and using lock-free or thread-safe data structure instead of lock at high-level, for example, a thread-safe b-tree could have a lock for each node instead of lock the whole tree.

[rleungx]

1. https://github.com/tikv/pd/blob/v6.3.0/server/cluster/cluster.go#L809 doesn't have to lock everything but could use a per region/per region bucket lock
2. https://github.com/tikv/pd/blob/v6.3.0/server/core/basic_cluster.go#L30, this locks 2 different data structures, could be separated.
3. or even better, BasicCluster could only lock thing based on its business needs, and using lock-free or thread-safe data structure instead of lock at high-level, for example, a thread-safe b-tree could have a lock for each node instead of lock the whole tree.

For 1, I think the purpose of this lock is that we wanna keep the atomicity of the whole progress of updating the cache. And both 2 and 3 are good choices.

[rleungx]

Related to #3557

[nolouch]

More details task list here: #5648

[rleungx]

Close it for now since #5648 is closed.