Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

通过Systemctl stop pd.service停止Loader PD服务，存在丢数据现象 #1146

Closed
jiancaiHub opened this issue Jul 11, 2018 · 10 comments
Closed

通过Systemctl stop pd.service停止Loader PD服务，存在丢数据现象 #1146

jiancaiHub opened this issue Jul 11, 2018 · 10 comments
Labels
type/question The issue belongs to a question.

Comments

@jiancaiHub
Copy link

jiancaiHub commented Jul 11, 2018 •
edited
Loading

部署方式：tidb-ansible
部署集群：
## TiDB Cluster Part
[tidb_servers]
172.20.94.37
[tikv_servers]
172.20.94.36
172.20.94.35
172.20.94.38
[pd_servers]
172.20.94.36
172.20.94.35
172.20.94.38
操作：

  1. 开始写数据（写数据32*100万）；
  2. stop leader；
  3. Monitoring；
  4. 统计32张表的Count
    现象：有若干张表的Count不足100万，更甚的丢失达40万数据
    日志：
2018/07/11 11:02:26.518 main.go:96: [info] Got signal [15] to exit.
2018/07/11 11:02:26.579 server.go:240: [info] closing server
2018/07/11 11:02:26.579 leader.go:278: [info] keep alive channel is closed
2018/07/11 11:02:26.579 coordinator.go:435: [info] label-scheduler stopped: context canceled
2018/07/11 11:02:26.584 coordinator.go:435: [info] balance-region-scheduler stopped: context canceled
2018/07/11 11:02:26.584 coordinator.go:435: [info] balance-leader-scheduler stopped: context canceled
2018/07/11 11:02:26.584 coordinator.go:435: [info] balance-hot-region-scheduler stopped: context canceled
2018/07/11 11:02:26.584 leader.go:74: [info] server is closed, return leader loop
2018/07/11 11:02:37.585 log.go:86: [info] etcdserver: [a9b1db5ae96ad5c3 starts leadership transfer from a9b1db5ae96ad5c3 to 9880df74a14a2857]
2018/07/11 11:02:37.585 log.go:86: [info] raft: [a9b1db5ae96ad5c3 [term 2] starts to transfer leadership to 9880df74a14a2857]
2018/07/11 11:02:37.585 log.go:86: [info] raft: [a9b1db5ae96ad5c3 sends MsgTimeoutNow to 9880df74a14a2857 immediately as 9880df74a14a2857 already has up-to-date log]
2018/07/11 11:02:37.587 log.go:86: [info] raft: [a9b1db5ae96ad5c3 [term: 2] received a MsgVote message with higher term from 9880df74a14a2857 [term: 3]]
2018/07/11 11:02:37.587 log.go:86: [info] raft: [a9b1db5ae96ad5c3 became follower at term 3]
2018/07/11 11:02:37.587 log.go:86: [info] raft: [a9b1db5ae96ad5c3 [logterm: 2, index: 1081, vote: 0] cast MsgVote for 9880df74a14a2857 [logterm: 2, index: 1081] at term 3]
2018/07/11 11:02:37.587 log.go:86: [info] raft: [raft.node: a9b1db5ae96ad5c3 lost leader a9b1db5ae96ad5c3 at term 3]
2018/07/11 11:02:37.590 log.go:86: [info] raft: [raft.node: a9b1db5ae96ad5c3 elected leader 9880df74a14a2857 at term 3]
2018/07/11 11:02:38.085 log.go:86: [info] etcdserver: [a9b1db5ae96ad5c3 finished leadership transfer from a9b1db5ae96ad5c3 to 9880df74a14a2857 (took 500.244497ms)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [stopping peer 9880df74a14a2857...]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [closed the TCP streaming connection with peer 9880df74a14a2857 (stream MsgApp v2 writer)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [stopped streaming with peer 9880df74a14a2857 (writer)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [closed the TCP streaming connection with peer 9880df74a14a2857 (stream Message writer)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [stopped streaming with peer 9880df74a14a2857 (writer)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [stopped HTTP pipelining with peer 9880df74a14a2857]
2018/07/11 11:02:38.086 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer 9880df74a14a2857 (stream MsgApp v2 reader)]
2018/07/11 11:02:38.086 log.go:86: [info] rafthttp: [stopped streaming with peer 9880df74a14a2857 (stream MsgApp v2 reader)]
2018/07/11 11:02:38.087 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer 9880df74a14a2857 (stream Message reader)]
2018/07/11 11:02:38.087 log.go:86: [info] rafthttp: [stopped streaming with peer 9880df74a14a2857 (stream Message reader)]
2018/07/11 11:02:38.087 log.go:86: [info] rafthttp: [stopped peer 9880df74a14a2857]
2018/07/11 11:02:38.087 log.go:86: [info] rafthttp: [stopping peer ccbc3a95c9b8d3df...]
2018/07/11 11:02:38.087 log.go:86: [info] rafthttp: [closed the TCP streaming connection with peer ccbc3a95c9b8d3df (stream MsgApp v2 writer)]
2018/07/11 11:02:38.087 log.go:86: [info] rafthttp: [stopped streaming with peer ccbc3a95c9b8d3df (writer)]
2018/07/11 11:02:38.094 log.go:86: [info] rafthttp: [closed the TCP streaming connection with peer ccbc3a95c9b8d3df (stream Message writer)]
2018/07/11 11:02:38.095 log.go:86: [info] rafthttp: [stopped peer ccbc3a95c9b8d3df]
2018/07/11 11:02:38.097 server.go:256: [info] close server
@disksing
Copy link
Contributor

Hi @jiancaiHub , table data does not store in PD server, so in theory kill PD leader won't cause data lost. Have you count tables before killing PD?

@disksing disksing reopened this Jul 11, 2018
@disksing disksing added the type/question The issue belongs to a question. label Jul 11, 2018
@jiancaiHub
Copy link
Author

jiancaiHub commented Jul 11, 2018 •
edited
Loading

@disksing
整个操作是为了验证PD宕机对写数据没影响。所以我使用了脚本来写32张表，每张表是100万条数据。

  1. 执行脚本（删除数据表并开始写数据）
  2. 在写数据的过程中，我停掉PD服务模拟宕机。与此同时数据写入仍然在执行
  3. 观察一段时间后，把PD服务拉起来（Leader已经重新选举，此时拉起的服务为Follower）
  4. 等写入数据的脚本运行完毕，统计结果。
  5. 有几张表的Count不满100万。

@disksing
Copy link
Contributor

When the PD leader is down, it will cause the cluster unavailable for a period of time. How do you import data? Did the importer report any errors when PD is killed? Did you confirm the return value of all statement executions during the import process and retry if necessary?

@jiancaiHub
Copy link
Author

@disksing
hi~
我使用TiDB给出的压测工具导入数据：https://github.com/pingcap/tidb-bench/blob/cwen/not_prepared_statement/sysbench/parallel-prepare.sh
我的理解：集群在Leader选举出来之前是不可用的，所以这段时间的插入语句会报错，会让用户感知到。

@disksing
Copy link
Contributor

hi @jiancaiHub , could you upload log files of TiDB and PD to help us investigate? If the log files are too long, you can first send logs when killing pd server and when querying total number of rows.

@jackysp
Copy link
Contributor

jackysp commented Jul 11, 2018

Hi @jiancaiHub ,
If you use sysbench 1.0 or above, please use its official prepare method. Your script is only for the old version of sysbench and maybe out-of-date.

@jiancaiHub
Copy link
Author

@disksing Leader掉线的过程
TiDB的日志

2018/07/11 11:02:37.144 session.go:488: [warning] [con:87] Retry:0 query:0 INSERT INTO sbtest15(k, c, pad) VALUES(7422493, '77590521772-41477593530-82823564161-18276499492-79044818863-41599671742-11668418761-95639766795-42157227480-02584524834', '67419692643-29303089181-54031327951-56610915633-24389556301'),(7422494, '06437987386-38007627053-57575123223-95855634770-62218371259-26030736606-44135535097-56420233988-77447833530-46513181175', '15696613227-80928163438-73294121617-96777962796-71765639919'),(7422495, '35240990699-14023685077-55745672052-29071622267-67204899418-83794180715-88139235648-53773156668-22167925495-45170710874', '28948784242-69891017676-64442279438-53098101176-61349718454'),(7422496, '65104519265-05347390378-29386639757-33285738661-98699674328-95090146187-80231621521-01881158721-40520778322-65171744518', '37816180019-39668165076-31817977026-49952173744-32214191564'),(7422497, '71604764751-50492685004-96352706042-02803116992-84183682491-19866000690-95079476794-18452344605-73101617078-64612729882', '87202641298-53585420037-89945076486-82822866424-60981992714'),(74224(len:524141)
2018/07/11 11:02:37.199 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.205 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.223 backoff.go:218: [warning] backoffer.maxSleep 5000ms is exceeded, errors:
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
2018/07/11 11:02:37.223 2pc.go:606: [warning] [con:93] 2PC get commitTS failed: [tikv:9001]PD server timeout[try again later], tid: 401415338339860481
2018/07/11 11:02:37.223 session.go:340: [warning] [con:93] retryable error: [tikv:9001]PD server timeout[try again later], txn: {<nil> <nil> 0xc4206f58e0 map[] [] <nil>}
2018/07/11 11:02:37.225 session.go:488: [warning] [con:93] Retry:0 query:0 INSERT INTO sbtest19(k, c, pad) VALUES(9203225, '48531957855-74794385943-22590051168-44532779422-57279868297-55242655230-96176790765-69475808849-95423072149-27526363376', '74236264429-74017161837-93438180220-80490984661-09960079419'),(9203226, '45275647384-37354021107-82174875585-91652388092-07471822858-21082827894-95220308793-23695340916-58516118903-58627610097', '99192075949-22732179281-49524016874-29507069162-06497528728'),(9203227, '15132043284-08260526997-18416085228-77899573319-58883563233-62601965827-91395196816-22421074001-15247441068-97667819487', '18317740479-98112093363-82817131779-83743595291-59655283658'),(9203228, '73300654065-05019264496-84052568486-27830257870-43040592173-65447858592-84801224897-58168355898-02254800098-92736616527', '53994107242-23494082642-94776561572-60042081867-75310576212'),(9203229, '39075176508-54190695587-68110811180-47183879548-23310750808-55520576884-59296531371-15113103353-40641256965-33228509416', '02061949600-40346163450-92103505457-69397021098-87448957031'),(92032(len:524141)
2018/07/11 11:02:37.231 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.246 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.252 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.273 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.305 backoff.go:218: [warning] backoffer.maxSleep 5000ms is exceeded, errors:
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
2018/07/11 11:02:37.305 txn.go:37: [error] [kv] RunInNewTxn error - [tikv:9001]PD server timeout[try again later]
2018/07/11 11:02:37.305 ddl_worker.go:69: [error] [ddl] handle ddl job err /home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/store/tikv/backoff.go:137: [tikv:9001]PD server timeout[try again later]
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/store/tikv/kv.go:301:
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/store/tikv/txn.go:53:
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/store/tikv/kv.go:240:
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/kv/txn.go:38:
/home/jenkins/workspace/build_tidb_2.0/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:311:
2018/07/11 11:02:37.328 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.352 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.376 client.go:297: [error] [pd] failed updateLeader: failed to get leader from [http://172.20.94.35:2379 http://172.20.94.36:2379 http://172.20.94.38:2379]
2018/07/11 11:02:37.394 backoff.go:218: [warning] backoffer.maxSleep 5000ms is exceeded, errors:
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable
get timestamp failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable

PD Follower的日志

2018/07/11 10:37:16.191 leader.go:95: [info] leader is name:"pd1" member_id:12227795646797239747 peer_urls:"http://172.20.94.36:2380" client_urls:"http://172.20.94.36:2379" , watch it
2018/07/11 11:02:30.815 leader.go:312: [info] leader is deleted
2018/07/11 11:02:30.817 leader.go:97: [info] leader changed, try to campaign leader
2018/07/11 11:02:30.817 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:31.018 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:31.219 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:31.420 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:31.622 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:31.823 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:32.024 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:32.226 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:32.428 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:32.640 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:32.841 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:33.042 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:33.243 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:33.445 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:33.646 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:33.847 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:34.048 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:34.251 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:34.452 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:34.653 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:34.854 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:35.056 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:35.257 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:35.458 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:35.659 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:35.861 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:36.062 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:36.264 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:36.465 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:36.667 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:36.868 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:37.074 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:37.275 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:37.476 leader.go:103: [info] pd3 is not etcd leader, skip campaign leader and check later
2018/07/11 11:02:37.587 log.go:86: [info] raft: [ccbc3a95c9b8d3df [term: 2] received a MsgVote message with higher term from 9880df74a14a2857 [term: 3]]
2018/07/11 11:02:37.588 log.go:86: [info] raft: [ccbc3a95c9b8d3df became follower at term 3]
2018/07/11 11:02:37.588 log.go:86: [info] raft: [ccbc3a95c9b8d3df [logterm: 2, index: 1081, vote: 0] cast MsgVote for 9880df74a14a2857 [logterm: 2, index: 1081] at term 3]
2018/07/11 11:02:37.588 log.go:86: [info] raft: [raft.node: ccbc3a95c9b8d3df lost leader a9b1db5ae96ad5c3 at term 3]
2018/07/11 11:02:37.590 log.go:86: [info] raft: [raft.node: ccbc3a95c9b8d3df elected leader 9880df74a14a2857 at term 3]
2018/07/11 11:02:38.087 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer a9b1db5ae96ad5c3 (stream MsgApp v2 reader)]
2018/07/11 11:02:38.092 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer a9b1db5ae96ad5c3 (stream Message reader)]
2018/07/11 11:02:38.188 log.go:80: [error] rafthttp: [failed to dial a9b1db5ae96ad5c3 on stream MsgApp v2 (dial tcp 172.20.94.36:2380: connect: connection refused)]
2018/07/11 11:02:38.188 log.go:86: [info] rafthttp: [peer a9b1db5ae96ad5c3 became inactive]
2018/07/11 11:02:43.356 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer a9b1db5ae96ad5c3 (stream MsgApp v2 writer)]
2018/07/11 11:02:43.357 log.go:82: [warning] rafthttp: [lost the TCP streaming connection with peer a9b1db5ae96ad5c3 (stream Message writer)]
2018/07/11 11:02:47.589 etcd_kv.go:116: [error] load from etcd error: context deadline exceeded
2018/07/11 11:02:47.589 etcd_kv.go:119: [warning] kv gets too slow: key /pd/6576783071050188848/member/14752730894367773663/leader_priority cost 10.002635276s err context deadline exceeded
2018/07/11 11:02:47.676 etcd_kv.go:116: [error] load from etcd error: context deadline exceeded
2018/07/11 11:02:47.676 etcd_kv.go:119: [warning] kv gets too slow: key /pd/6576783071050188848/leader cost 10.000494309s err context deadline exceeded
2018/07/11 11:02:47.677 leader.go:80: [error] get leader err context deadline exceeded
2018/07/11 11:02:48.587 log.go:82: [warning] etcdserver: [timed out waiting for read index response]
2018/07/11 11:02:48.588 leader.go:95: [info] leader is name:"pd2" member_id:10989028782799202391 peer_urls:"http://172.20.94.35:2380" client_urls:"http://172.20.94.35:2379" , watch it
2018/07/11 11:02:50.072 log.go:82: [warning] rafthttp: [health check for peer a9b1db5ae96ad5c3 could not connect: dial tcp 172.20.94.36:2380: connect: connection refused]

@jiancaiHub
Copy link
Author

@jackysp

sysbench -v
sysbench 1.0.9 (using system LuaJIT 2.0.4)

@jackysp
Copy link
Contributor

jackysp commented Jul 12, 2018

sysbench --threads=32 oltp_point_select --tables=32 --table-size=1000000 prepare
Please use the above command for your test.

@disksing
Copy link
Contributor

Close it due to no feedback for a long time. Please feel free to reopen it if need.

ti-chi-bot pushed a commit that referenced this issue Feb 15, 2023
@dependabot
build(deps): bump github.com/prometheus/client_golang from 1.11.0 to …
3a286c2 
…1.11.1 in /tools/pd-tso-bench (#5990)

ref #897, ref #962, ref #969, ref #974, ref #975, ref #976, ref #986, ref prometheus/client_golang#987, ref #987, ref #989, ref #998, ref #1013, ref #1014, ref #1025, ref #1028, ref #1031, ref #1043, ref #1055, ref #1075, ref #1091, ref #1094, ref #1102, ref #1103, ref #1118, ref #1146, ref #1148, ref #1150

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to lhy1024/pd that referenced this issue Feb 15, 2023
@dependabot
build(deps): bump github.com/prometheus/client_golang from 1.11.0 to …
e168bd6 
…1.11.1 in /tests/client (tikv#5992)

ref tikv#897, ref tikv#962, ref tikv#969, ref tikv#974, ref tikv#975, ref tikv#976, ref tikv#986, ref tikv#987, ref prometheus/client_golang#987, ref tikv#989, ref tikv#998, ref tikv#1013, ref tikv#1014, ref tikv#1025, ref tikv#1028, ref tikv#1031, ref tikv#1043, ref tikv#1055, ref tikv#1075, ref tikv#1091, ref tikv#1094, ref tikv#1102, ref tikv#1103, ref tikv#1118, ref tikv#1146, ref tikv#1148, ref tikv#1150, ref tikv#4399

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ti-chi-bot added a commit that referenced this issue Feb 15, 2023
@dependabot @ti-chi-bot
build(deps): bump github.com/prometheus/client_golang from 1.11.0 to …
2839786 
…1.11.1 in /tests/mcs (#5993)

ref #897, ref #962, ref #969, ref #974, ref #975, ref #976, ref #986, ref #987, ref prometheus/client_golang#987, ref #989, ref #998, ref #1013, ref #1014, ref #1025, ref #1028, ref #1031, ref #1043, ref #1055, ref #1075, ref #1091, ref #1094, ref #1102, ref #1103, ref #1118, ref #1146, ref #1148, ref #1150, ref #4399

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ti-chi-bot added a commit that referenced this issue Feb 15, 2023
@dependabot @ti-chi-bot
build(deps): bump github.com/prometheus/client_golang from 1.11.0 to …
951dbc6 
…1.11.1 in /client (#5991)

ref #897, ref #962, ref #969, ref #974, ref #975, ref #976, ref #986, ref #987, ref prometheus/client_golang#987, ref #989, ref #998, ref #1013, ref #1014, ref #1025, ref #1028, ref #1031, ref #1043, ref #1055, ref #1075, ref #1091, ref #1094, ref #1102, ref #1103, ref #1118, ref #1146, ref #1148, ref #1150, ref #4399

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/question The issue belongs to a question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jackysp @disksing @jiancaiHub