Issue with loading TPC-C warehouses

[Yan-Ou]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• Deploy tidb cluster respectively on c6g.2xlarge instance's and r5b.2xlarge instance's EKS clusters.
• Load TPC-C 5000 wareshouses into the clusters as per the instructions here: https://docs.pingcap.com/zh/tidb/stable/benchmark-tidb-using-tpcc
• Watch the EKS nodes status

watch kubectl get node

You will see the nodes become NotReady status. If you ssh into the nodes before the loading process and top the resource utilisation, you would see the memory get eating up quickly.

• After setting resource limit for tibd and tikv pods, the nodes and pods would continue to function. Despite the loading completed eventually, there were missing data when running the benchmark test:

23:05:54,548 [main] INFO   jTPCC : Term-00,                                                                                       23:06:27,491 [Thread-0] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER 21MB / 483MB
23:06:27,497 [Thread-0] ERROR  jTPCCTData : District for W_ID=88 D_ID=10 not found
java.sql.SQLException: District for W_ID=88 D_ID=10 not found
        at jTPCCTData.executeNewOrder(jTPCCTData.java:366)
        at jTPCCTData.execute(jTPCCTData.java:95)
        at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:277)
        at jTPCCTerminal.run(jTPCCTerminal.java:88)
        at java.lang.Thread.run(Thread.java:748)                                                                                  23:06:27,566 [Thread-15] FATAL  jTPCCTerminal : District for W_ID=568 D_ID=8 not found20MB / 483MB
java.lang.Exception: District for W_ID=568 D_ID=8 not found
        at jTPCCTData.executePayment(jTPCCTData.java:777)
        at jTPCCTData.execute(jTPCCTData.java:99)
        at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:160)
        at jTPCCTerminal.run(jTPCCTerminal.java:88)
        at java.lang.Thread.run(Thread.java:748

• During the loading, there were many error messages:

Worker 012: ERROR: null
java.lang.NullPointerException
        at com.mysql.cj.protocol.a.NativeProtocol.clearInputStream(NativeProtocol.java:826)
        at com.mysql.cj.NativeSession.clearInputStream(NativeSession.java:337)
        at com.mysql.cj.jdbc.ServerPreparedStatement.serverPrepare(ServerPreparedStatement.java:660)
        at com.mysql.cj.jdbc.ServerPreparedStatement.<init>(ServerPreparedStatement.java:133)
        at com.mysql.cj.jdbc.ServerPreparedStatement.getInstance(ServerPreparedStatement.java:102)
        at com.mysql.cj.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:1632)
        at com.mysql.cj.jdbc.ServerPreparedStatement.prepareBatchedInsertSQL(ServerPreparedStatement.java:790)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchedInserts(ClientPreparedStatement.java:711)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:447)
        at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:814)
        at LoadDataWorker.loadWarehouse(LoadDataWorker.java:401)
        at LoadDataWorker.run(LoadDataWorker.java:194)
        at java.lang.Thread.run(Thread.java:748)
Worker 400: Loading Warehouse    400

Worker 606: ERROR: Region is unavailable
Worker 689: ERROR: Region is unavailable
Worker 514: ERROR: Region is unavailable
Worker 685: ERROR: Region is unavailable
Worker 688: ERROR: Region is unavailable
Worker 532: ERROR: Region is unavailable
[2021/03/16 00:19:01.391 +00:00] [WARN] [2pc.go:885] ["send TxnHeartBeat failed"] [error="txn 423586410320887851 heartbeat fail, primary key = 74800000000000003f5f69800000000000000103800000000000010e038000000000000007038000000000000835, err = txn 423586410320887851 not found"] [txnStartTS=423586410320887851]

2. What did you expect to see? (Required)

I would expect to see data loading and validating complete successfully.

3. What did you see instead (Required)

Loading either failed or does not complete.

4. What is your TiDB version? (Required)

v5.0.0-rc for both x86 and arm

[sticnarf]

TiDB has been improving its flow control and retry mechanisms from TiDB 5.1 to 5.3, so it is now unlikely to produce these Region is unavailable errors.

I would recommend you to try the recent versions. Feel free to open new issues if you encounter the errors again on TiDB 5.2+. Thanks!

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.