cooridinator: fix the issue caused adeadlock when deleting scheduler (#2625)

[ti-srebot]

cherry-pick #2625 to release-4.0

---

Signed-off-by: nolouch nolouch@gmail.com

What problem does this PR solve?

The PD service cannot serve after remove the scheduler.

What is changed and how it works?

The main reason:

goroutine 1:

• To delete the schedule, call `func (c *RaftCluster) RemoveScheduler(name string)
• Hold RaftCluster write lock
• Wait for coordinator write lock

goroutine 2:

• Background thread, collect statistics continuously, call func (c *coordinator) collectHotSpotMetrics()
• Hold the coordinator read lock
• Call s.Scheduler.(hasHotStatus).GetHotWriteStatus() to wait for hotScheduler read lock

goroutine3:

• Scheduling process, scheduling, call func (h *hotScheduler) Schedule(cluster opt.Cluster)
• Hold hotScheduler write lock
• Call cluster.GetStoresStats() to wait for RaftCluster read lock

Check List

• Manual test (add detailed scripts or steps below)

Release note

• fix the issue that caused deadlock when deleting scheduler

[ti-srebot]

@nolouch please accept the invitation then you can push to the cherry-pick pull requests.
https://github.com/ti-srebot/pd/invitations

[lhy1024]

LGTM

[lhy1024]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@ti-srebot merge failed.

[lhy1024]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 2638

[ti-srebot]

/run-all-tests

[ti-srebot]

@ti-srebot merge failed.

[lhy1024]

/merge

[ti-srebot]

/run-all-tests

[codecov-commenter]

Codecov Report

Merging #2637 into release-4.0 will decrease coverage by 0.00%.
The diff coverage is 50.00%.

@@               Coverage Diff               @@
##           release-4.0    #2637      +/-   ##
===============================================
- Coverage        77.16%   77.16%   -0.01%     
===============================================
  Files              205      205              
  Lines            22115    22116       +1     
===============================================
- Hits             17066    17065       -1     
  Misses            3747     3747              
- Partials          1302     1304       +2     

Impacted Files	Coverage Δ
server/cluster/coordinator.go	74.31% <50.00%> (-2.40%)	⬇️
server/kv/etcd_kv.go	84.41% <0.00%> (-3.90%)	⬇️
server/member/leader.go	71.20% <0.00%> (-3.12%)	⬇️
server/schedulers/random_merge.go	61.19% <0.00%> (-2.99%)	⬇️
server/region_syncer/client.go	84.73% <0.00%> (-1.53%)	⬇️
server/schedule/operator_controller.go	81.41% <0.00%> (+0.16%)	⬆️
server/server.go	75.86% <0.00%> (+0.45%)	⬆️
server/grpc_service.go	59.84% <0.00%> (+1.15%)	⬆️
pkg/dashboard/adapter/manager.go	86.79% <0.00%> (+2.83%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 999331e...2f43afc. Read the comment docs.

[HunDunDM]

LGTM

[lhy1024]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 2640
• 2635
• 2636

[ti-srebot]

/run-all-tests