Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink: Kafka message length and batch size limit for Open Protocol (#1079) #1112

Merged
merged 1 commit into from
Nov 24, 2020
Merged

sink: Kafka message length and batch size limit for Open Protocol (#1079) #1112

merged 1 commit into from
Nov 24, 2020

Conversation

ti-srebot
Copy link
Contributor

cherry-pick #1079 to release-4.0

What problem does this PR solve?

  • Open Protocol messages risk exceeding the message length limit for Kafka.
  • Users cannot control the number of records in a single message produced by Open Protocol.

What is changed and how it works?

  • The Open Protocol Encoder reads the max-message-bytes parameter from the sink-uri and respects this limit.
  • The Open Protocol Encoder now reads max-batch-size from opts and respects this limit.

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has interface methods change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release note

  • No release note

@liuzix @ti-srebot
cherry pick pingcap#1079 to release-4.0
4249bcf 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot
Copy link
Contributor Author

/run-all-tests

@ti-srebot ti-srebot mentioned this pull request Nov 24, 2020
Merged
@ti-srebot ti-srebot added component/open-protocol Open TiCDC protocol component. component/sink Sink component. type/4.0-cherry-pick labels Nov 24, 2020
@ti-srebot ti-srebot added this to the v4.0.9 milestone Nov 24, 2020
@zier-one
Copy link
Contributor

/lgtm

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 24, 2020
@zier-one
Copy link
Contributor

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 24, 2020
@ti-srebot
Copy link
Contributor Author

/run-all-tests

@ti-srebot
Copy link
Contributor Author

@ti-srebot merge failed.

@zier-one
Copy link
Contributor

/run-kafka-tests

@zier-one zier-one merged commit a643787 into pingcap:release-4.0 Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3pointer 3pointer Awaiting requested review from 3pointer

@amyangfei amyangfei Awaiting requested review from amyangfei

@zier-one zier-one Awaiting requested review from zier-one

Assignees

@liuzix liuzix

Labels
component/open-protocol Open TiCDC protocol component. component/sink Sink component. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Milestone
v4.0.9
Development

Successfully merging this pull request may close these issues.
3 participants
@ti-srebot @zier-one @liuzix