Length of messages written to Kafka exceeds max-message-bytes

[liuzix]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   Set message-max-bytes parameter in sink-uri to a number less than 4MB, when using the Kafka sink.

2. What did you expect to see?
   That replication works normally.

3. What did you see instead?
   That CDC exits with error CDC:ErrKafkaAsyncSendMessage

Analysis

This bug is caused by mqSink not limiting the message size according to message-max-bytes, which is only used by KafkaProducer but not mqSink. mqSink should make use of this value, and avoid issuing writes to Kafka that are large than message-max-bytes.

Caveats

• What if we have a change record that exceeds message-max-bytes even on its own?
• Protocols such as Canal produce messages whose sizes are not easily predictable based on sizes of individual records and are not known until a complete message has been produced. What is the best way to determine it is time to cut off the message?

[amyangfei]

fixed by #1079