scheduler: scheduled the all tikv pods to same node

[xiaojingchen]

after deploy tidbcluster, found all tikv pods were scheduled to one node.

[tidb@172.16.30.59 ~]$ kubectl get po -n example3 -o wide
NAME                                  READY     STATUS      RESTARTS   AGE       IP              NODE
example3-monitor-configurator-cnhkd   0/1       Completed   0          7m        10.128.35.166   172.16.30.118
example3-monitor-d85c86ccc-vmk2s      2/2       Running     0          7m        10.128.32.186   172.16.30.59
example3-pd-0                         1/1       Running     0          7m        10.128.34.78    172.16.30.98
example3-pd-1                         1/1       Running     0          7m        10.128.35.187   172.16.30.118
example3-pd-2                         1/1       Running     0          7m        10.128.47.59    172.16.30.58
example3-tidb-0                       1/1       Running     0          6m        10.128.42.214   172.16.30.216
example3-tidb-1                       1/1       Running     0          6m        10.128.35.185   172.16.30.118
example3-tidb-initializer-mfjxt       0/1       Completed   0          7m        10.128.36.4     172.16.30.117
example3-tikv-0                       2/2       Running     1          7m        10.128.34.66    172.16.30.98
example3-tikv-1                       2/2       Running     0          7m        10.128.34.121   172.16.30.98
example3-tikv-2                       2/2       Running     0          7m        10.128.34.90    172.16.30.98

the tidb-scheduler log:

I0111 10:19:22.367319       1 scheduler.go:75] scheduling pod: example2/example2-pd-0
I0111 10:19:22.367365       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.216 172.16.30.58 172.16.30.98 172.16.30.117 172.16.30.59 172.16.30.118]
I0111 10:19:22.379762       1 ha.go:103] []
I0111 10:19:22.379799       1 ha.go:104] nodeMap: map[172.16.30.58:[] 172.16.30.98:[] 172.16.30.117:[] 172.16.30.59:[] 172.16.30.118:[] 172.16.30.216:[]]
I0111 10:19:22.379832       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.216 172.16.30.58 172.16.30.98 172.16.30.117 172.16.30.59 172.16.30.118]
I0111 10:19:23.479056       1 scheduler.go:75] scheduling pod: example2/example2-pd-0
I0111 10:19:23.479093       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.118]
I0111 10:19:23.482692       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.118]
I0111 10:19:36.871942       1 scheduler.go:75] scheduling pod: example2/example2-pd-1
I0111 10:19:36.872005       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.59 172.16.30.98 172.16.30.216 172.16.30.117 172.16.30.58 172.16.30.118]
I0111 10:19:36.879774       1 ha.go:103] [172.16.30.118 ]
I0111 10:19:36.879799       1 ha.go:104] nodeMap: map[172.16.30.59:[] 172.16.30.98:[] 172.16.30.216:[] 172.16.30.117:[] 172.16.30.58:[] 172.16.30.118:[example2-pd-0]]
I0111 10:19:36.879822       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.59 172.16.30.98 172.16.30.216 172.16.30.117 172.16.30.58]
I0111 10:19:37.898867       1 scheduler.go:75] scheduling pod: example2/example2-pd-1
I0111 10:19:37.898916       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.98]
I0111 10:19:37.902729       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.98]
I0111 10:19:47.171127       1 scheduler.go:75] scheduling pod: example2/example2-pd-2
I0111 10:19:47.171157       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.58 172.16.30.59 172.16.30.98 172.16.30.216 172.16.30.117 172.16.30.118]
I0111 10:19:47.180153       1 ha.go:103] [172.16.30.118 172.16.30.98 ]
I0111 10:19:47.180172       1 ha.go:104] nodeMap: map[172.16.30.58:[] 172.16.30.59:[] 172.16.30.98:[example2-pd-1] 172.16.30.216:[] 172.16.30.117:[] 172.16.30.118:[example2-pd-0]]
I0111 10:19:47.180211       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.58 172.16.30.59 172.16.30.216 172.16.30.117]
I0111 10:19:47.269610       1 scheduler.go:75] scheduling pod: example2/example2-tikv-0
I0111 10:19:47.269691       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.58 172.16.30.216 172.16.30.117 172.16.30.98 172.16.30.59 172.16.30.118]
I0111 10:19:47.281870       1 ha.go:103] [  ]
I0111 10:19:47.281900       1 ha.go:104] nodeMap: map[172.16.30.58:[] 172.16.30.216:[] 172.16.30.117:[] 172.16.30.98:[] 172.16.30.59:[] 172.16.30.118:[]]
I0111 10:19:47.281938       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.58 172.16.30.216 172.16.30.117 172.16.30.98 172.16.30.59 172.16.30.118]
I0111 10:19:47.468379       1 scheduler.go:75] scheduling pod: example2/example2-tikv-1
I0111 10:19:47.468409       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.118 172.16.30.58 172.16.30.117 172.16.30.216 172.16.30.59 172.16.30.98]
I0111 10:19:47.477768       1 ha.go:103] [  ]
I0111 10:19:47.477786       1 ha.go:104] nodeMap: map[172.16.30.118:[] 172.16.30.58:[] 172.16.30.117:[] 172.16.30.216:[] 172.16.30.59:[] 172.16.30.98:[]]
I0111 10:19:47.477810       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.118 172.16.30.58 172.16.30.117 172.16.30.216 172.16.30.59 172.16.30.98]
I0111 10:19:47.864180       1 scheduler.go:75] scheduling pod: example2/example2-tikv-2
I0111 10:19:47.864213       1 scheduler.go:78] entering predicate: HighAvailability, nodes: [172.16.30.98 172.16.30.59 172.16.30.216 172.16.30.117 172.16.30.58 172.16.30.118]
I0111 10:19:47.873935       1 ha.go:103] [  ]
I0111 10:19:47.873971       1 ha.go:104] nodeMap: map[172.16.30.118:[] 172.16.30.98:[] 172.16.30.59:[] 172.16.30.216:[] 172.16.30.117:[] 172.16.30.58:[]]
I0111 10:19:47.873995       1 scheduler.go:83] leaving predicate: HighAvailability, nodes: [172.16.30.98 172.16.30.59 172.16.30.216 172.16.30.117 172.16.30.58 172.16.30.118]
I0111 10:19:48.197416       1 scheduler.go:75] scheduling pod: example2/example2-pd-2

found that the pod's spec.nodeName is empty, it may be set after some time. this cause tidb-scheduler ha filter invalid.

[weekface]

So the kube-scheduler is parallel scheduling, we need a solution to make sure only one pod can be scheduled at the same time.

[weekface]

Need a lock to do this.

Can't use sync.Mutex as the lock, because the tidb-scheduler can crash.
Can't use TidbCluster as the lock also, because the TidbCluster may be deleted.

@xiaojingchen suggest to use PVC as the lock, it is related to the real data(PV), and its life cycle is longer than pod.