Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scatter region is unexpected which causes uneven region distribution #5909

Closed
rleungx opened this issue Feb 6, 2023 · 0 comments · Fixed by #5920
Closed

Scatter region is unexpected which causes uneven region distribution #5909

rleungx opened this issue Feb 6, 2023 · 0 comments · Fixed by #5920
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-6.6 severity/minor type/bug The issue is confirmed as a bug.

Comments

@rleungx
Copy link
Member

rleungx commented Feb 6, 2023 •
edited
Loading

Bug Report

What did you do?

Do scatter region operation.

What did you expect to see?

The region distributed evenly among stores

What did you see instead?

[2023/02/03 07:02:02.345 +00:00] [INFO] [cluster_worker.go:145] ["alloc ids for region split"] [region-id=127106] [peer-ids="[127107,127108,127109]"]
[2023/02/03 07:02:02.372 +00:00] [INFO] [cluster_worker.go:237] ["region batch split, generate new regions"] [region-id=23] [origin="id:127046 start_key:\"7480000000000000FFA95F72D00000001AFF825AAC0000000
000FA\" end_key:\"7480000000000000FFA95F72D00000001BFF7B846E0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127047 store_id:4 > peers:<id:127048 store_id:8 > peers:<id:127049 store_id:6
 > id:127050 start_key:\"7480000000000000FFA95F72D00000001BFF7B846E0000000000FA\" end_key:\"7480000000000000FFA95F72D00000001CFF8260FB0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127
051 store_id:4 > peers:<id:127052 store_id:8 > peers:<id:127053 store_id:6 > id:127054 start_key:\"7480000000000000FFA95F72D00000001CFF8260FB0000000000FA\" end_key:\"7480000000000000FFA95F72D800000000FF0F
854E0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127055 store_id:4 > peers:<id:127056 store_id:8 > peers:<id:127057 store_id:6 > id:127058 start_key:\"7480000000000000FFA95F72D800000
000FF0F854E0000000000FA\" end_key:\"7480000000000000FFA95F72D800000001FF183AAB0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127059 store_id:4 > peers:<id:127060 store_id:8 > peers:<id
:127061 store_id:6 > id:127062 start_key:\"7480000000000000FFA95F72D800000001FF183AAB0000000000FA\" end_key:\"7480000000000000FFA95F72D800000002FF4DC6C60000000000FA\" region_epoch:<conf_ver:23 version:150
19 > peers:<id:127063 store_id:4 > peers:<id:127064 store_id:8 > peers:<id:127065 store_id:6 > id:127066 start_key:\"7480000000000000FFA95F72D800000002FF4DC6C60000000000FA\" end_key:\"7480000000000000FFA9
5F72D800000003FF4E38750000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127067 store_id:4 > peers:<id:127068 store_id:8 > peers:<id:127069 store_id:6 > id:127070 start_key:\"7480000000000
000FFA95F72D800000003FF4E38750000000000FA\" end_key:\"7480000000000000FFA95F72D800000004FF7DDD0B0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127071 store_id:4 > peers:<id:127072 stor
e_id:8 > peers:<id:127073 store_id:6 > id:127074 start_key:\"7480000000000000FFA95F72D800000004FF7DDD0B0000000000FA\" end_key:\"7480000000000000FFA95F72D800000005FF9817F90000000000FA\" region_epoch:<conf_
ver:23 version:15019 > peers:<id:127075 store_id:4 > peers:<id:127076 store_id:8 > peers:<id:127077 store_id:6 > id:127078 start_key:\"7480000000000000FFA95F72D800000005FF9817F90000000000FA\" end_key:\"74
80000000000000FFA95F72D800000006FFD1E7ED0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127079 store_id:4 > peers:<id:127080 store_id:8 > peers:<id:127081 store_id:6 > id:127082 start_k
ey:\"7480000000000000FFA95F72D800000006FFD1E7ED0000000000FA\" end_key:\"7480000000000000FFA95F72D800000007FFFBFB830000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127083 store_id:4 > pee
rs:<id:127084 store_id:8 > peers:<id:127085 store_id:6 > id:127086 start_key:\"7480000000000000FFA95F72D800000007FFFBFB830000000000FA\" end_key:\"7480000000000000FFA95F72D800000009FF3F6F070000000000FA\" r
egion_epoch:<conf_ver:23 version:15019 > peers:<id:127087 store_id:4 > peers:<id:127088 store_id:8 > peers:<id:127089 store_id:6 > id:127090 start_key:\"7480000000000000FFA95F72D800000009FF3F6F07000000000
0FA\" end_key:\"7480000000000000FFA95F72D80000000AFF5C138E0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127091 store_id:4 > peers:<id:127092 store_id:8 > peers:<id:127093 store_id:6 >
 id:127094 start_key:\"7480000000000000FFA95F72D80000000AFF5C138E0000000000FA\" end_key:\"7480000000000000FFA95F72D80000000BFF800F050000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:12709
5 store_id:4 > peers:<id:127096 store_id:8 > peers:<id:127097 store_id:6 > id:127098 start_key:\"7480000000000000FFA95F72D80000000BFF800F050000000000FA\" end_key:\"7480000000000000FFA95F72D80000000CFF8675
C10000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127099 store_id:4 > peers:<id:127100 store_id:8 > peers:<id:127101 store_id:6 > id:127102 start_key:\"7480000000000000FFA95F72D80000000
CFF8675C10000000000FA\" end_key:\"7480000000000000FFA95F72D80000000DFF9A58570000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127103 store_id:4 > peers:<id:127104 store_id:8 > peers:<id:1
27105 store_id:6 > id:127106 start_key:\"7480000000000000FFA95F72D80000000DFF9A58570000000000FA\" end_key:\"7480000000000000FFA95F72D80000000EFFE1742F0000000000FA\" region_epoch:<conf_ver:23 version:15019
 > peers:<id:127107 store_id:4 > peers:<id:127108 store_id:8 > peers:<id:127109 store_id:6 > id:127110 start_key:\"7480000000000000FFA95F72D80000000EFFE1742F0000000000FA\" end_key:\"7480000000000000FFA95F
72D800000010FF1D661F0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127111 store_id:4 > peers:<id:127112 store_id:8 > peers:<id:127113 store_id:6 > id:127114 start_key:\"748000000000000
0FFA95F72D800000010FF1D661F0000000000FA\" end_key:\"7480000000000000FFA95F72D800000011FF494C510000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127115 store_id:4 > peers:<id:127116 store_
id:8 > peers:<id:127117 store_id:6 > id:127118 start_key:\"7480000000000000FFA95F72D800000011FF494C510000000000FA\" end_key:\"7480000000000000FFA95F72D800000012FF61242C0000000000FA\" region_epoch:<conf_ve
r:23 version:15019 > peers:<id:127119 store_id:4 > peers:<id:127120 store_id:8 > peers:<id:127121 store_id:6 > id:127122 start_key:\"7480000000000000FFA95F72D800000012FF61242C0000000000FA\" end_key:\"7480
000000000000FFA95F72D800000013FF6F282F0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127123 store_id:4 > peers:<id:127124 store_id:8 > peers:<id:127125 store_id:6 > id:127126 start_key
:\"7480000000000000FFA95F72D800000013FF6F282F0000000000FA\" end_key:\"7480000000000000FFA95F72D800000014FF988DF50000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127127 store_id:4 > peers
:<id:127128 store_id:8 > peers:<id:127129 store_id:6 > id:127130 start_key:\"7480000000000000FFA95F72D800000014FF988DF50000000000FA\" end_key:\"7480000000000000FFA95F72D800000015FFDDF6730000000000FA\" reg
ion_epoch:<conf_ver:23 version:15019 > peers:<id:127131 store_id:4 > peers:<id:127132 store_id:8 > peers:<id:127133 store_id:6 > id:127134 start_key:\"7480000000000000FFA95F72D800000015FFDDF6730000000000F
A\" end_key:\"7480000000000000FFA95F72D800000016FFEBD0E30000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127135 store_id:4 > peers:<id:127136 store_id:8 > peers:<id:127137 store_id:6 > i
d:127138 start_key:\"7480000000000000FFA95F72D800000016FFEBD0E30000000000FA\" end_key:\"7480000000000000FFA95F72D800000018FF257D4C0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127139
store_id:4 > peers:<id:127140 store_id:8 > peers:<id:127141 store_id:6 > id:127142 start_key:\"7480000000000000FFA95F72D800000018FF257D4C0000000000FA\" end_key:\"7480000000000000FFA95F72D800000019FF5D0DE7
0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127143 store_id:4 > peers:<id:127144 store_id:8 > peers:<id:127145 store_id:6 > id:127146 start_key:\"7480000000000000FFA95F72D800000019F
F5D0DE70000000000FA\" end_key:\"7480000000000000FFA95F72D80000001AFF67E3320000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127147 store_id:4 > peers:<id:127148 store_id:8 > peers:<id:127
149 store_id:6 > id:127150 start_key:\"7480000000000000FFA95F72D80000001AFF67E3320000000000FA\" end_key:\"7480000000000000FFA95F72D80000001BFF99E7540000000000FA\" region_epoch:<conf_ver:23 version:15019 >
 peers:<id:127151 store_id:4 > peers:<id:127152 store_id:8 > peers:<id:127153 store_id:6 > id:127154 start_key:\"7480000000000000FFA95F72D80000001BFF99E7540000000000FA\" end_key:\"7480000000000000FFA95F72
D80000001CFFB0DF170000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127155 store_id:4 > peers:<id:127156 store_id:8 > peers:<id:127157 store_id:6 > id:127158 start_key:\"7480000000000000F
FA95F72D80000001CFFB0DF170000000000FA\" end_key:\"7480000000000000FFA95F72E000000000FF2FBD900000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127159 store_id:4 > peers:<id:127160 store_id
:8 > peers:<id:127161 store_id:6 > id:127162 start_key:\"7480000000000000FFA95F72E000000000FF2FBD900000000000FA\" end_key:\"7480000000000000FFA95F72E000000001FF7D91920000000000FA\" region_epoch:<conf_ver:
23 version:15019 > peers:<id:127163 store_id:4 > peers:<id:127164 store_id:8 > peers:<id:127165 store_id:6 > id:127166 start_key:\"7480000000000000FFA95F72E000000001FF7D91920000000000FA\" end_key:\"748000
0000000000FFA95F72E000000002FFBE89320000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127167 store_id:4 > peers:<id:127168 store_id:8 > peers:<id:127169 store_id:6 > id:127170 start_key:\
"7480000000000000FFA95F72E000000002FFBE89320000000000FA\" end_key:\"7480000000000000FFA95F72E000000003FFC219AB0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127171 store_id:4 > peers:<
id:127172 store_id:8 > peers:<id:127173 store_id:6 > id:127174 start_key:\"7480000000000000FFA95F72E000000003FFC219AB0000000000FA\" end_key:\"7480000000000000FFA95F72E000000004FFEEB2C80000000000FA\" regio
n_epoch:<conf_ver:23 version:15019 > peers:<id:127175 store_id:4 > peers:<id:127176 store_id:8 > peers:<id:127177 store_id:6 > id:127178 start_key:\"7480000000000000FFA95F72E000000004FFEEB2C80000000000FA\
" end_key:\"7480000000000000FFA95F72E000000006FF39803A0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127179 store_id:4 > peers:<id:127180 store_id:8 > peers:<id:127181 store_id:6 > id:
127182 start_key:\"7480000000000000FFA95F72E000000006FF39803A0000000000FA\" end_key:\"7480000000000000FFA95F72E000000007FF78B82A0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127183 st
ore_id:4 > peers:<id:127184 store_id:8 > peers:<id:127185 store_id:6 > id:127186 start_key:\"7480000000000000FFA95F72E000000007FF78B82A0000000000FA\" end_key:\"7480000000000000FFA95F72E000000008FF901A2200
00000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127187 store_id:4 > peers:<id:127188 store_id:8 > peers:<id:127189 store_id:6 > id:127190 start_key:\"7480000000000000FFA95F72E000000008FF9
01A220000000000FA\" end_key:\"7480000000000000FFA95F72E000000009FF713C850000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127191 store_id:4 > peers:<id:127192 store_id:8 > peers:<id:12719
3 store_id:6 > id:127194 start_key:\"7480000000000000FFA95F72E000000009FF713C850000000000FA\" end_key:\"7480000000000000FFA95F72E00000000AFF8BAC710000000000FA\" region_epoch:<conf_ver:23 version:15019 > p
eers:<id:127195 store_id:4 > peers:<id:127196 store_id:8 > peers:<id:127197 store_id:6 > id:127198 start_key:\"7480000000000000FFA95F72E00000000AFF8BAC710000000000FA\" end_key:\"7480000000000000FFA95F72E0
0000000BFF9B8FFE0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127199 store_id:4 > peers:<id:127200 store_id:8 > peers:<id:127201 store_id:6 > id:127202 start_key:\"7480000000000000FFA
95F72E00000000BFF9B8FFE0000000000FA\" end_key:\"7480000000000000FFA95F72E00000000CFFB46EA40000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127203 store_id:4 > peers:<id:127204 store_id:8
 > peers:<id:127205 store_id:6 > id:127206 start_key:\"7480000000000000FFA95F72E00000000CFFB46EA40000000000FA\" end_key:\"7480000000000000FFA95F72E00000000DFFF686CF0000000000FA\" region_epoch:<conf_ver:23
 version:15019 > peers:<id:127207 store_id:4 > peers:<id:127208 store_id:8 > peers:<id:127209 store_id:6 > id:127210 start_key:\"7480000000000000FFA95F72E00000000DFFF686CF0000000000FA\" end_key:\"74800000
00000000FFA95F72E00000000EFFFE4F6E0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127211 store_id:4 > peers:<id:127212 store_id:8 > peers:<id:127213 store_id:6 > id:127214 start_key:\"7
480000000000000FFA95F72E00000000EFFFE4F6E0000000000FA\" end_key:\"7480000000000000FFA95F72E000000010FF24B6D40000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127215 store_id:4 > peers:<id
:127216 store_id:8 > peers:<id:127217 store_id:6 > id:127218 start_key:\"7480000000000000FFA95F72E000000010FF24B6D40000000000FA\" end_key:\"7480000000000000FFA95F72E000000011FF59FBFD0000000000FA\" region_
epoch:<conf_ver:23 version:15019 > peers:<id:127219 store_id:4 > peers:<id:127220 store_id:8 > peers:<id:127221 store_id:6 > id:127222 start_key:\"7480000000000000FFA95F72E000000011FF59FBFD0000000000FA\"
end_key:\"7480000000000000FFA95F72E000000012FFAD7C380000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127223 store_id:4 > peers:<id:127224 store_id:8 > peers:<id:127225 store_id:6 > id:12
7226 start_key:\"7480000000000000FFA95F72E000000012FFAD7C380000000000FA\" end_key:\"7480000000000000FFA95F72E000000013FFF068770000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127227 stor
e_id:4 > peers:<id:127228 store_id:8 > peers:<id:127229 store_id:6 > id:127230 start_key:\"7480000000000000FFA95F72E000000013FFF068770000000000FA\" end_key:\"7480000000000000FFA95F72E000000014FFF447140000
000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127231 store_id:4 > peers:<id:127232 store_id:8 > peers:<id:127233 store_id:6 > id:127234 start_key:\"7480000000000000FFA95F72E000000014FFF44
7140000000000FA\" end_key:\"7480000000000000FFA95F72E000000016FF44359F0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127235 store_id:4 > peers:<id:127236 store_id:8 > peers:<id:127237
store_id:6 > id:127238 start_key:\"7480000000000000FFA95F72E000000016FF44359F0000000000FA\" end_key:\"7480000000000000FFA95F72E000000017FF4785E30000000000FA\" region_epoch:<conf_ver:23 version:15019 > pee
rs:<id:127239 store_id:4 > peers:<id:127240 store_id:8 > peers:<id:127241 store_id:6 > id:127242 start_key:\"7480000000000000FFA95F72E000000017FF4785E30000000000FA\" end_key:\"7480000000000000FFA95F72E000
000018FF6284E80000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127243 store_id:4 > peers:<id:127244 store_id:8 > peers:<id:127245 store_id:6 > id:127246 start_key:\"7480000000000000FFA95
F72E000000018FF6284E80000000000FA\" end_key:\"7480000000000000FFA95F72E000000019FF8E3EB30000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127247 store_id:4 > peers:<id:127248 store_id:8 >
 peers:<id:127249 store_id:6 > id:127250 start_key:\"7480000000000000FFA95F72E000000019FF8E3EB30000000000FA\" end_key:\"7480000000000000FFA95F72E00000001AFFBF0D970000000000FA\" region_epoch:<conf_ver:23 v
ersion:15019 > peers:<id:127251 store_id:4 > peers:<id:127252 store_id:8 > peers:<id:127253 store_id:6 > id:127254 start_key:\"7480000000000000FFA95F72E00000001AFFBF0D970000000000FA\" end_key:\"7480000000
000000FFA95F72E00000001BFFE5C0B10000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127255 store_id:4 > peers:<id:127256 store_id:8 > peers:<id:127257 store_id:6 > id:127258 start_key:\"748
0000000000000FFA95F72E00000001BFFE5C0B10000000000FA\" end_key:\"7480000000000000FFA95F72E00000001DFF0BF4320000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127259 store_id:4 > peers:<id:1
27260 store_id:8 > peers:<id:127261 store_id:6 > id:127262 start_key:\"7480000000000000FFA95F72E00000001DFF0BF4320000000000FA\" end_key:\"7480000000000000FFA95F72E800000000FF7BBE590000000000FA\" region_ep
och:<conf_ver:23 version:15019 > peers:<id:127263 store_id:4 > peers:<id:127264 store_id:8 > peers:<id:127265 store_id:6 > id:127266 start_key:\"7480000000000000FFA95F72E800000000FF7BBE590000000000FA\" en
d_key:\"7480000000000000FFA95F72E800000001FF96C3090000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127267 store_id:4 > peers:<id:127268 store_id:8 > peers:<id:127269 store_id:6 > id:1272
70 start_key:\"7480000000000000FFA95F72E800000001FF96C3090000000000FA\" end_key:\"7480000000000000FFA95F72E800000002FFA5E1A50000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127271 store_
id:4 > peers:<id:127272 store_id:8 > peers:<id:127273 store_id:6 > id:127274 start_key:\"7480000000000000FFA95F72E800000002FFA5E1A50000000000FA\" end_key:\"7480000000000000FFA95F72E800000003FFC6AD7E000000
0000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127275 store_id:4 > peers:<id:127276 store_id:8 > peers:<id:127277 store_id:6 > id:127278 start_key:\"7480000000000000FFA95F72E800000003FFC6AD7
E0000000000FA\" end_key:\"7480000000000000FFA95F72E800000005FF06F3F70000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127279 store_id:4 > peers:<id:127280 store_id:8 > peers:<id:127281 st
ore_id:6 > id:127282 start_key:\"7480000000000000FFA95F72E800000005FF06F3F70000000000FA\" end_key:\"7480000000000000FFA95F72E800000006FF3843AA0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers
:<id:127283 store_id:4 > peers:<id:127284 store_id:8 > peers:<id:127285 store_id:6 > id:127286 start_key:\"7480000000000000FFA95F72E800000006FF3843AA0000000000FA\" end_key:\"7480000000000000FFA95F72E80000
0007FF78BA8C0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127287 store_id:4 > peers:<id:127288 store_id:8 > peers:<id:127289 store_id:6 > id:127290 start_key:\"7480000000000000FFA95F7
2E800000007FF78BA8C0000000000FA\" end_key:\"7480000000000000FFA95F72E800000008FF8FCB570000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127291 store_id:4 > peers:<id:127292 store_id:8 > p
eers:<id:127293 store_id:6 > id:127294 start_key:\"7480000000000000FFA95F72E800000008FF8FCB570000000000FA\" end_key:\"7480000000000000FFA95F72E800000009FFBE16C30000000000FA\" region_epoch:<conf_ver:23 ver
sion:15019 > peers:<id:127295 store_id:4 > peers:<id:127296 store_id:8 > peers:<id:127297 store_id:6 > id:127298 start_key:\"7480000000000000FFA95F72E800000009FFBE16C30000000000FA\" end_key:\"748000000000
0000FFA95F72E80000000AFFD3E9D80000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127299 store_id:4 > peers:<id:127300 store_id:8 > peers:<id:127301 store_id:6 > id:127302 start_key:\"74800
00000000000FFA95F72E80000000AFFD3E9D80000000000FA\" end_key:\"7480000000000000FFA95F72E80000000BFFDED4720000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127303 store_id:4 > peers:<id:127
304 store_id:8 > peers:<id:127305 store_id:6 > id:127306 start_key:\"7480000000000000FFA95F72E80000000BFFDED4720000000000FA\" end_key:\"7480000000000000FFA95F72E80000000CFFEF35760000000000FA\" region_epoc
h:<conf_ver:23 version:15019 > peers:<id:127307 store_id:4 > peers:<id:127308 store_id:8 > peers:<id:127309 store_id:6 > id:127310 start_key:\"7480000000000000FFA95F72E80000000CFFEF35760000000000FA\" end_
key:\"7480000000000000FFA95F72E80000000EFF04394E0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127311 store_id:4 > peers:<id:127312 store_id:8 > peers:<id:127313 store_id:6 > id:127314
 start_key:\"7480000000000000FFA95F72E80000000EFF04394E0000000000FA\" end_key:\"7480000000000000FFA95F72E80000000FFF4917510000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127315 store_id
:4 > peers:<id:127316 store_id:8 > peers:<id:127317 store_id:6 > id:127318 start_key:\"7480000000000000FFA95F72E80000000FFF4917510000000000FA\" end_key:\"7480000000000000FFA95F72E800000010FF4E143F00000000
00FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127319 store_id:4 > peers:<id:127320 store_id:8 > peers:<id:127321 store_id:6 > id:127322 start_key:\"7480000000000000FFA95F72E800000010FF4E143F0
000000000FA\" end_key:\"7480000000000000FFA95F72E800000011FF684C500000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127323 store_id:4 > peers:<id:127324 store_id:8 > peers:<id:127325 stor
e_id:6 > id:127326 start_key:\"7480000000000000FFA95F72E800000011FF684C500000000000FA\" end_key:\"7480000000000000FFA95F72E800000012FF8387DD0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<
id:127327 store_id:4 > peers:<id:127328 store_id:8 > peers:<id:127329 store_id:6 > id:127330 start_key:\"7480000000000000FFA95F72E800000012FF8387DD0000000000FA\" end_key:\"7480000000000000FFA95F72E8000000
13FFBD31CC0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127331 store_id:4 > peers:<id:127332 store_id:8 > peers:<id:127333 store_id:6 > id:127334 start_key:\"7480000000000000FFA95F72E
800000013FFBD31CC0000000000FA\" end_key:\"7480000000000000FFA95F72E800000014FFCF16D70000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127335 store_id:4 > peers:<id:127336 store_id:8 > pee
rs:<id:127337 store_id:6 > id:127338 start_key:\"7480000000000000FFA95F72E800000014FFCF16D70000000000FA\" end_key:\"7480000000000000FFA95F72E800000015FFCE7C920000000000FA\" region_epoch:<conf_ver:23 versi
on:15019 > peers:<id:127339 store_id:4 > peers:<id:127340 store_id:8 > peers:<id:127341 store_id:6 > id:127342 start_key:\"7480000000000000FFA95F72E800000015FFCE7C920000000000FA\" end_key:\"74800000000000
00FFA95F72E800000016FFCE8B930000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127343 store_id:4 > peers:<id:127344 store_id:8 > peers:<id:127345 store_id:6 > id:127346 start_key:\"7480000
000000000FFA95F72E800000016FFCE8B930000000000FA\" end_key:\"7480000000000000FFA95F72E800000017FFE565BB0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127347 store_id:4 > peers:<id:12734
8 store_id:8 > peers:<id:127349 store_id:6 > id:127350 start_key:\"7480000000000000FFA95F72E800000017FFE565BB0000000000FA\" end_key:\"7480000000000000FFA95F72E800000019FF1433290000000000FA\" region_epoch:
<conf_ver:23 version:15019 > peers:<id:127351 store_id:4 > peers:<id:127352 store_id:8 > peers:<id:127353 store_id:6 > id:127354 start_key:\"7480000000000000FFA95F72E800000019FF1433290000000000FA\" end_ke
y:\"7480000000000000FFA95F72E80000001AFF5DC6860000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127355 store_id:4 > peers:<id:127356 store_id:8 > peers:<id:127357 store_id:6 > id:127358 s
tart_key:\"7480000000000000FFA95F72E80000001AFF5DC6860000000000FA\" end_key:\"7480000000000000FFA95F72E80000001BFF89F3250000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127359 store_id:4
 > peers:<id:127360 store_id:8 > peers:<id:127361 store_id:6 > id:127362 start_key:\"7480000000000000FFA95F72E80000001BFF89F3250000000000FA\" end_key:\"7480000000000000FFA95F72E80000001CFF92566E0000000000
FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127363 store_id:4 > peers:<id:127364 store_id:8 > peers:<id:127365 store_id:6 > id:127366 start_key:\"7480000000000000FFA95F72E80000001CFF92566E000
0000000FA\" end_key:\"7480000000000000FFA95F72F000000000FF3B8F8C0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127367 store_id:4 > peers:<id:127368 store_id:8 > peers:<id:127369 store_
id:6 > id:127370 start_key:\"7480000000000000FFA95F72F000000000FF3B8F8C0000000000FA\" end_key:\"7480000000000000FFA95F72F000000001FF61047A0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id
:127371 store_id:4 > peers:<id:127372 store_id:8 > peers:<id:127373 store_id:6 > id:127374 start_key:\"7480000000000000FFA95F72F000000001FF61047A0000000000FA\" end_key:\"7480000000000000FFA95F72F000000002
FF77C3E50000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127375 store_id:4 > peers:<id:127376 store_id:8 > peers:<id:127377 store_id:6 > id:127378 start_key:\"7480000000000000FFA95F72F00
0000002FF77C3E50000000000FA\" end_key:\"7480000000000000FFA95F72F000000003FF94415D0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127379 store_id:4 > peers:<id:127380 store_id:8 > peers
:<id:127381 store_id:6 > id:127382 start_key:\"7480000000000000FFA95F72F000000003FF94415D0000000000FA\" end_key:\"7480000000000000FFA95F72F000000004FFA7C3240000000000FA\" region_epoch:<conf_ver:23 version
:15019 > peers:<id:127383 store_id:4 > peers:<id:127384 store_id:8 > peers:<id:127385 store_id:6 > id:127386 start_key:\"7480000000000000FFA95F72F000000004FFA7C3240000000000FA\" end_key:\"7480000000000000
FFA95F72F000000005FFA5C8F00000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127387 store_id:4 > peers:<id:127388 store_id:8 > peers:<id:127389 store_id:6 > id:127390 start_key:\"748000000
0000000FFA95F72F000000005FFA5C8F00000000000FA\" end_key:\"7480000000000000FFA95F72F000000006FFC93AF00000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127391 store_id:4 > peers:<id:127392
store_id:8 > peers:<id:127393 store_id:6 > id:127394 start_key:\"7480000000000000FFA95F72F000000006FFC93AF00000000000FA\" end_key:\"7480000000000000FFA95F72F000000007FFD13E580000000000FA\" region_epoch:<c
onf_ver:23 version:15019 > peers:<id:127395 store_id:4 > peers:<id:127396 store_id:8 > peers:<id:127397 store_id:6 > id:127398 start_key:\"7480000000000000FFA95F72F000000007FFD13E580000000000FA\" end_key:
\"7480000000000000FFA95F72F000000008FFD5F0230000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127399 store_id:4 > peers:<id:127400 store_id:8 > peers:<id:127401 store_id:6 > id:127402 sta
rt_key:\"7480000000000000FFA95F72F000000008FFD5F0230000000000FA\" end_key:\"7480000000000000FFA95F72F000000009FFFB24FD0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127403 store_id:4 >
 peers:<id:127404 store_id:8 > peers:<id:127405 store_id:6 > id:127406 start_key:\"7480000000000000FFA95F72F000000009FFFB24FD0000000000FA\" end_key:\"7480000000000000FFA95F72F00000000BFF3A001E0000000000FA
\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127407 store_id:4 > peers:<id:127408 store_id:8 > peers:<id:127409 store_id:6 > id:127410 start_key:\"7480000000000000FFA95F72F00000000BFF3A001E00000
00000FA\" end_key:\"7480000000000000FFA95F72F00000000CFF49A7F50000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127411 store_id:4 > peers:<id:127412 store_id:8 > peers:<id:127413 store_id
:6 > id:127414 start_key:\"7480000000000000FFA95F72F00000000CFF49A7F50000000000FA\" end_key:\"7480000000000000FFA95F72F00000000DFF52F7A40000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:1
27415 store_id:4 > peers:<id:127416 store_id:8 > peers:<id:127417 store_id:6 > id:127418 start_key:\"7480000000000000FFA95F72F00000000DFF52F7A40000000000FA\" end_key:\"7480000000000000FFA95F72F00000000EFF
948B420000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127419 store_id:4 > peers:<id:127420 store_id:8 > peers:<id:127421 store_id:6 > id:127422 start_key:\"7480000000000000FFA95F72F0000
0000EFF948B420000000000FA\" end_key:\"7480000000000000FFA95F72F00000000FFFA5C7140000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127423 store_id:4 > peers:<id:127424 store_id:8 > peers:<
id:127425 store_id:6 > id:127426 start_key:\"7480000000000000FFA95F72F00000000FFFA5C7140000000000FA\" end_key:\"7480000000000000FFA95F72F000000010FFA694B20000000000FA\" region_epoch:<conf_ver:23 version:1
5019 > peers:<id:127427 store_id:4 > peers:<id:127428 store_id:8 > peers:<id:127429 store_id:6 > id:127430 start_key:\"7480000000000000FFA95F72F000000010FFA694B20000000000FA\" end_key:\"7480000000000000FF
A95F72F000000011FFC884760000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127431 store_id:4 > peers:<id:127432 store_id:8 > peers:<id:127433 store_id:6 > id:127434 start_key:\"74800000000
00000FFA95F72F000000011FFC884760000000000FA\" end_key:\"7480000000000000FFA95F72F000000012FFDCAA910000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127435 store_id:4 > peers:<id:127436 st
ore_id:8 > peers:<id:127437 store_id:6 > id:127438 start_key:\"7480000000000000FFA95F72F000000012FFDCAA910000000000FA\" end_key:\"7480000000000000FFA95F72F000000013FFEBA3550000000000FA\" region_epoch:<con
f_ver:23 version:15019 > peers:<id:127439 store_id:4 > peers:<id:127440 store_id:8 > peers:<id:127441 store_id:6 > id:127442 start_key:\"7480000000000000FFA95F72F000000013FFEBA3550000000000FA\" end_key:\"
7480000000000000FFA95F72F000000015FF1445B30000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127443 store_id:4 > peers:<id:127444 store_id:8 > peers:<id:127445 store_id:6 > id:127446 start
_key:\"7480000000000000FFA95F72F000000015FF1445B30000000000FA\" end_key:\"7480000000000000FFA95F72F000000016FF2E3C850000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127447 store_id:4 > p
eers:<id:127448 store_id:8 > peers:<id:127449 store_id:6 > id:127450 start_key:\"7480000000000000FFA95F72F000000016FF2E3C850000000000FA\" end_key:\"7480000000000000FFA95F72F000000017FF71CF8D0000000000FA\"
 region_epoch:<conf_ver:23 version:15019 > peers:<id:127451 store_id:4 > peers:<id:127452 store_id:8 > peers:<id:127453 store_id:6 > id:127454 start_key:\"7480000000000000FFA95F72F000000017FF71CF8D0000000
000FA\" end_key:\"7480000000000000FFA95F72F000000018FF7271BA0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127455 store_id:4 > peers:<id:127456 store_id:8 > peers:<id:127457 store_id:6
 > id:127458 start_key:\"7480000000000000FFA95F72F000000018FF7271BA0000000000FA\" end_key:\"7480000000000000FFA95F72F000000019FF7F06990000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127
459 store_id:4 > peers:<id:127460 store_id:8 > peers:<id:127461 store_id:6 > id:127462 start_key:\"7480000000000000FFA95F72F000000019FF7F06990000000000FA\" end_key:\"7480000000000000FFA95F72F00000001AFFB0
38CF0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127463 store_id:4 > peers:<id:127464 store_id:8 > peers:<id:127465 store_id:6 > id:127466 start_key:\"7480000000000000FFA95F72F000000
01AFFB038CF0000000000FA\" end_key:\"7480000000000000FFA95F72F00000001BFFDC0E990000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127467 store_id:4 > peers:<id:127468 store_id:8 > peers:<id
:127469 store_id:6 > id:127470 start_key:\"7480000000000000FFA95F72F00000001BFFDC0E990000000000FA\" end_key:\"7480000000000000FFA95F72F00000001CFFEE07330000000000FA\" region_epoch:<conf_ver:23 version:150
19 > peers:<id:127471 store_id:4 > peers:<id:127472 store_id:8 > peers:<id:127473 store_id:6 > id:127474 start_key:\"7480000000000000FFA95F72F00000001CFFEE07330000000000FA\" end_key:\"7480000000000000FFA9
5F72F800000000FF6A44DA0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127475 store_id:4 > peers:<id:127476 store_id:8 > peers:<id:127477 store_id:6 > id:127478 start_key:\"7480000000000
000FFA95F72F800000000FF6A44DA0000000000FA\" end_key:\"7480000000000000FFA95F72F800000001FF7C79DC0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127479 store_id:4 > peers:<id:127480 stor
e_id:8 > peers:<id:127481 store_id:6 > id:127482 start_key:\"7480000000000000FFA95F72F800000001FF7C79DC0000000000FA\" end_key:\"7480000000000000FFA95F72F800000002FFA342F00000000000FA\" region_epoch:<conf_
ver:23 version:15019 > peers:<id:127483 store_id:4 > peers:<id:127484 store_id:8 > peers:<id:127485 store_id:6 > id:127486 start_key:\"7480000000000000FFA95F72F800000002FFA342F00000000000FA\" end_key:\"74
80000000000000FFA95F72F800000003FFC7FE570000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127487 store_id:4 > peers:<id:127488 store_id:8 > peers:<id:127489 store_id:6 > id:127490 start_k
ey:\"7480000000000000FFA95F72F800000003FFC7FE570000000000FA\" end_key:\"7480000000000000FFA95F72F800000004FFFD37E10000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127491 store_id:4 > pee
rs:<id:127492 store_id:8 > peers:<id:127493 store_id:6 > id:127494 start_key:\"7480000000000000FFA95F72F800000004FFFD37E10000000000FA\" end_key:\"7480000000000000FFA95F72F800000005FFEEFA900000000000FA\" r
egion_epoch:<conf_ver:23 version:15019 > peers:<id:127495 store_id:4 > peers:<id:127496 store_id:8 > peers:<id:127497 store_id:6 > id:127498 start_key:\"7480000000000000FFA95F72F800000005FFEEFA90000000000
0FA\" end_key:\"7480000000000000FFA95F72F800000007FF05D35F0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127499 store_id:4 > peers:<id:127500 store_id:8 > peers:<id:127501 store_id:6 >
 id:127502 start_key:\"7480000000000000FFA95F72F800000007FF05D35F0000000000FA\" end_key:\"7480000000000000FFA95F72F800000008FF253FF80000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:12750
3 store_id:4 > peers:<id:127504 store_id:8 > peers:<id:127505 store_id:6 > id:127506 start_key:\"7480000000000000FFA95F72F800000008FF253FF80000000000FA\" end_key:\"7480000000000000FFA95F72F800000009FF2C87
590000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127507 store_id:4 > peers:<id:127508 store_id:8 > peers:<id:127509 store_id:6 > id:127510 start_key:\"7480000000000000FFA95F72F80000000
9FF2C87590000000000FA\" end_key:\"7480000000000000FFA95F72F80000000AFF7C9B7C0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127511 store_id:4 > peers:<id:127512 store_id:8 > peers:<id:1
27513 store_id:6 > id:127514 start_key:\"7480000000000000FFA95F72F80000000AFF7C9B7C0000000000FA\" end_key:\"7480000000000000FFA95F72F80000000BFFB41B810000000000FA\" region_epoch:<conf_ver:23 version:15019
 > peers:<id:127515 store_id:4 > peers:<id:127516 store_id:8 > peers:<id:127517 store_id:6 > id:127518 start_key:\"7480000000000000FFA95F72F80000000BFFB41B810000000000FA\" end_key:\"7480000000000000FFA95F
72F80000000CFFC4F7B00000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127519 store_id:4 > peers:<id:127520 store_id:8 > peers:<id:127521 store_id:6 > id:127522 start_key:\"748000000000000
0FFA95F72F80000000CFFC4F7B00000000000FA\" end_key:\"7480000000000000FFA95F72F80000000DFFED30880000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127523 store_id:4 > peers:<id:127524 store_
id:8 > peers:<id:127525 store_id:6 > id:127526 start_key:\"7480000000000000FFA95F72F80000000DFFED30880000000000FA\" end_key:\"7480000000000000FFA95F72F80000000EFFE5CCB40000000000FA\" region_epoch:<conf_ve
r:23 version:15019 > peers:<id:127527 store_id:4 > peers:<id:127528 store_id:8 > peers:<id:127529 store_id:6 > id:127530 start_key:\"7480000000000000FFA95F72F80000000EFFE5CCB40000000000FA\" end_key:\"7480
000000000000FFA95F72F800000010FF1111990000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127531 store_id:4 > peers:<id:127532 store_id:8 > peers:<id:127533 store_id:6 > id:127534 start_key
:\"7480000000000000FFA95F72F800000010FF1111990000000000FA\" end_key:\"7480000000000000FFA95F72F800000011FF2C98130000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127535 store_id:4 > peers
:<id:127536 store_id:8 > peers:<id:127537 store_id:6 > id:127538 start_key:\"7480000000000000FFA95F72F800000011FF2C98130000000000FA\" end_key:\"7480000000000000FFA95F72F800000012FF530AA40000000000FA\" reg
ion_epoch:<conf_ver:23 version:15019 > peers:<id:127539 store_id:4 > peers:<id:127540 store_id:8 > peers:<id:127541 store_id:6 > id:127542 start_key:\"7480000000000000FFA95F72F800000012FF530AA40000000000F
A\" end_key:\"7480000000000000FFA95F72F800000013FF7C25F60000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127543 store_id:4 > peers:<id:127544 store_id:8 > peers:<id:127545 store_id:6 > i
d:127546 start_key:\"7480000000000000FFA95F72F800000013FF7C25F60000000000FA\" end_key:\"7480000000000000FFA95F72F800000014FF89E1520000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127547
store_id:4 > peers:<id:127548 store_id:8 > peers:<id:127549 store_id:6 > id:127550 start_key:\"7480000000000000FFA95F72F800000014FF89E1520000000000FA\" end_key:\"7480000000000000FFA95F72F800000015FFBF3A11
0000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127551 store_id:4 > peers:<id:127552 store_id:8 > peers:<id:127553 store_id:6 > id:127554 start_key:\"7480000000000000FFA95F72F800000015F
FBF3A110000000000FA\" end_key:\"7480000000000000FFA95F72F800000016FFF8FBB50000000000FA\" region_epoch:<conf_ver:23 version:15019 > peers:<id:127555 store_id:4 > peers:<id:127556 store_id:8 > peers:<id:127
557 store_id:6 >"] [total=128]
[2023/02/03 07:02:02.477 +00:00] [INFO] [operator_controller.go:452] ["add operator"] [region-id=127106] [operator="\"scatter-region {add peer: store [10 5]} (kind:admin,region,leader, region:127106(15019
, 23), createAt:2023-02-03 07:02:02.477110937 +0000 UTC m=+2653.357034447, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[add learner peer 127574 on store 5, add learner peer 127573
on store 10, transfer leader from store 4 to store 6, use joint consensus, promote learner peer 127574 on store 5 to voter, promote learner peer 127573 on store 10 to voter, leave joint state, promote lea
rner peer 127574 on store 5 to voter, promote learner peer 127573 on store 10 to voter]，timeout:[27m0s])\""] [additional-info=]
[2023/02/03 07:02:02.477 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="add learner peer 127574 on store 5"] [source=create]
[2023/02/03 07:02:02.495 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Add peer:{id:127574 store_id:5 role:Learner }"] [old-confver=23] [new-confver=24]
[2023/02/03 07:02:02.495 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="add learner peer 127574 on store 5"] [source=heartbeat]
[2023/02/03 07:02:02.532 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="add learner peer 127573 on store 10"] [source=heartbeat]
[2023/02/03 07:02:02.552 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Add peer:{id:127573 store_id:10 role:Learner }"] [old-confver=24] [new-confver=25]
[2023/02/03 07:02:02.552 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="add learner peer 127573 on store 10"] [source=heartbeat]
[2023/02/03 07:02:02.583 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="transfer leader from store 4 to store 6"] [source=heartbeat]
[2023/02/03 07:02:02.588 +00:00] [INFO] [region.go:652] ["leader changed"] [region-id=127106] [from=4] [to=6]
[2023/02/03 07:02:02.588 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:02.591 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:02.591 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:02.594 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:02.594 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:07.683 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="use joint consensus, promote learner peer 127574 on store 5 to voter, promote learn
er peer 127573 on store 10 to voter"] [source="active push"]
[2023/02/03 07:02:07.699 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Remove peer:{id:127574 store_id:5 role:Learner },Remove peer:{id:127573 store_id:10 role:Lear
ner },Add peer:{id:127574 store_id:5 role:IncomingVoter },Add peer:{id:127573 store_id:10 role:IncomingVoter }"] [old-confver=25] [new-confver=27]
[2023/02/03 07:02:07.699 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="leave joint state, promote learner peer 127574 on store 5 to voter, promote learner
 peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:07.699 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="leave joint state, promote learner peer 127574 on store 5 to voter, promote learner
 peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:07.699 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="leave joint state, promote learner peer 127574 on store 5 to voter, promote learner
 peer 127573 on store 10 to voter"] [source=heartbeat]
[2023/02/03 07:02:07.716 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Remove peer:{id:127574 store_id:5 role:IncomingVoter },Remove peer:{id:127573 store_id:10 rol
e:IncomingVoter },Add peer:{id:127574 store_id:5 },Add peer:{id:127573 store_id:10 }"] [old-confver=27] [new-confver=29]
[2023/02/03 07:02:07.716 +00:00] [INFO] [operator_controller.go:565] ["operator finish"] [region-id=127106] [takes=5.239322332s] [operator="\"scatter-region {add peer: store [10 5]} (kind:admin,region,lea
der, region:127106(15019, 23), createAt:2023-02-03 07:02:02.477110937 +0000 UTC m=+2653.357034447, startAt:2023-02-03 07:02:02.477148838 +0000 UTC m=+2653.357072348, currentStep:5, size:1, steps:[add lear
ner peer 127574 on store 5, add learner peer 127573 on store 10, transfer leader from store 4 to store 6, use joint consensus, promote learner peer 127574 on store 5 to voter, promote learner peer 127573
on store 10 to voter, leave joint state, promote learner peer 127574 on store 5 to voter, promote learner peer 127573 on store 10 to voter]，timeout:[27m0s]) finished\""] [additional-info=]
[2023/02/03 07:02:11.769 +00:00] [INFO] [operator_controller.go:452] ["add operator"] [region-id=127106] [operator="\"remove-orphan-peer {rm peer: store [10]} (kind:region, region:127106(15019, 29), creat
eAt:2023-02-03 07:02:11.769913113 +0000 UTC m=+2662.649836623, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[remove peer on store 10]，timeout:[1m0s])\""] [additional-info=]
[2023/02/03 07:02:11.770 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="remove peer on store 10"] [source=create]
[2023/02/03 07:02:11.792 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Remove peer:{id:127573 store_id:10 }"] [old-confver=29] [new-confver=30]
[2023/02/03 07:02:11.792 +00:00] [INFO] [operator_controller.go:565] ["operator finish"] [region-id=127106] [takes=22.100221ms] [operator="\"remove-orphan-peer {rm peer: store [10]} (kind:region, region:1
27106(15019, 29), createAt:2023-02-03 07:02:11.769913113 +0000 UTC m=+2662.649836623, startAt:2023-02-03 07:02:11.770046215 +0000 UTC m=+2662.649969825, currentStep:1, size:1, steps:[remove peer on store
10]，timeout:[1m0s]) finished\""] [additional-info=]
[2023/02/03 07:02:15.899 +00:00] [INFO] [operator_controller.go:452] ["add operator"] [region-id=127106] [operator="\"remove-orphan-peer {rm peer: store [5]} (kind:region, region:127106(15019, 30), create
At:2023-02-03 07:02:15.899047941 +0000 UTC m=+2666.778971551, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[remove peer on store 5]，timeout:[1m0s])\""] [additional-info=]
[2023/02/03 07:02:15.899 +00:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=127106] [step="remove peer on store 5"] [source=create]
[2023/02/03 07:02:15.912 +00:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=127106] [detail="Remove peer:{id:127574 store_id:5 }"] [old-confver=30] [new-confver=31]
[2023/02/03 07:02:15.912 +00:00] [INFO] [operator_controller.go:565] ["operator finish"] [region-id=127106] [takes=13.336694ms] [operator="\"remove-orphan-peer {rm peer: store [5]} (kind:region, region:12
7106(15019, 30), createAt:2023-02-03 07:02:15.899047941 +0000 UTC m=+2666.778971551, startAt:2023-02-03 07:02:15.899172443 +0000 UTC m=+2666.779096053, currentStep:1, size:1, steps:[remove peer on store 5
]，timeout:[1m0s]) finished\""] [additional-info=]

Here is all log related to region 127106. After splitting, there are 3 replicas located in store 4, 8, and 6. Then PD adds a scatter operator which adds two more replicas on store 5 and 10. But there is no more step. After a while, the rule checker found there are 5 replicas for region 127106 and it exceeds the expected replica number. So it removes the newly added replicas located in store 5, 10.

What version of PD are you using (pd-server -V)?

v6.5 but might affect many versions after v6.1, not sure for now.
@rleungx rleungx added the type/bug The issue is confirmed as a bug. label Feb 6, 2023
@HunDunDM HunDunDM mentioned this issue Feb 7, 2023
Merged
@HunDunDM HunDunDM added severity/minor affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Feb 7, 2023
ti-chi-bot pushed a commit that referenced this issue Feb 7, 2023
@HunDunDM
region_scatterer: fix the bug that could generate schedule with too m…
f5b5391 
…any peers (#5920)

ref #4570, close #5909

Signed-off-by: HunDunDM <hundundm@gmail.com>
This was referenced Feb 7, 2023
Merged
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
45bfc74 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 7, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
d2c5888 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 7, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
fec8cbb 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 7, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
368f7c7 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 7, 2023
Closed
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
30ee90f 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 7, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 7, 2023
@HunDunDM @ti-chi-bot
This is an automated cherry-pick of tikv#5920
eaf86ca 
ref tikv#4570, close tikv#5909

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot added a commit that referenced this issue Feb 13, 2023
@ti-chi-bot @HunDunDM
region_scatterer: fix the bug that could generate schedule with too m…
b04f0a9 
…any peers (#5920) (#5923)

ref #4570, close #5909, ref #5920

Signed-off-by: HunDunDM <hundundm@gmail.com>

Co-authored-by: HunDunDM <hundundm@gmail.com>
@nolouch nolouch mentioned this issue Mar 7, 2023
Merged
16 tasks
ti-chi-bot added a commit that referenced this issue Mar 27, 2023
@ti-chi-bot @HunDunDM @CabinfeverB
region_scatterer: fix the bug that could generate schedule with too m…
0646a37 
…any peers (#5920) (#5929)

ref #4570, close #5909, ref #5920

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Co-authored-by: 混沌DM <hundundm@gmail.com>
Co-authored-by: Cabinfever_B <cabinfeveroier@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.0 affects-5.1 affects-5.2 affects-5.3 affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 affects-6.3 affects-6.4 affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-6.6 severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

region_scatterer: fix the bug that could generate schedule with too many peers HunDunDM/pd
2 participants
@HunDunDM @rleungx