ticdc: send boot strap cost too many times leading to stuck the owner

[hongyunyan]

In user scenarios we find the owner is stuck for a long time. And in the logs, we find owner is stuck because too many time costed in send bootstrap messages, which is not excepted.

[2024/08/28 07:03:11.269 +00:00] [INFO] [ddl_manager.go:189] ["send bootstrap messages finished"] [changefeed=default/ingestion-test-4] [cost=3m1.504269564s]

[3AceShowHand]

"[2024/08/28 07:00:09.766 +00:00] [INFO] [ddl_manager.go:198] [\"start to send bootstrap messages\"] [changefeed=default/ingestion-test-4] [tables=923]",

[2024/08/28 07:03:11.269 +00:00] [INFO] [ddl_manager.go:189] ["send bootstrap messages finished"] [changefeed=default/ingestion-test-4] [cost=3m1.504269564s]

At the moment, bootstrap message is delivered one by one synchronously, it takes around 181 seconds to send 923 message, each one cost around 0.19s

[3AceShowHand]

[2024/09/06 11:36:09.379 +08:00] [INFO] [ddl_manager.go:190] ["start to send bootstrap messages"] [changefeed=default/simple-avro-test-checksum] [tables=100]
[2024/09/06 11:36:09.480 +08:00] [INFO] [ddl_manager.go:181] ["send bootstrap messages finished"] [changefeed=default/simple-avro-test-checksum] [cost=101.495145ms]

According to my local test, send 100 bootstrap message cost 101ms, 1.01ms for each one.

[3AceShowHand]

[2024/09/06 12:20:57.781 +08:00] [INFO] [ddl_manager.go:190] ["start to send bootstrap messages"] [changefeed=default/simple-avro-test-checksum] [tables=500]
[2024/09/06 12:20:58.538 +08:00] [INFO] [ddl_manager.go:181] ["send bootstrap messages finished"] [changefeed=default/simple-avro-test-checksum] [cost=757.550372ms]

500 tables, 757 ms

[seiya-annie]

/report customer