Incubating Program: Add Apache Pulsar to tidb-binlog #117
Labels
incubating-request/feature
This is an Incubating Program request for a feature
Comments
|
PTAL @GregoryIan |
|
@wolfstudy Thanks for your request. Could you please write an RFC or Proposal to https://github.com/pingcap/community/tree/master/rfc, and add a link in the description of this issue? |
winkyao
added
incubating-request/project
|
I will take a look |
|
@winkyao @GregoryIan @siddontang any update for this? |
|
ping @winkyao @GregoryIan @siddontang PTAL thanks. |
|
ping @winkyao @GregoryIan @siddontang any update for this? |
|
LGTM |
|
@GregoryIan @siddontang PTAL again, thanks |
IANTHEREAL
changed the title
IANTHEREAL
changed the title
|
I suggest integrating Apache Pulsar into ticdc, are there any issues for it? @wolfstudy |
rleungx
pushed a commit
to rleungx/community
that referenced
this issue
Jun 17, 2021
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
|
Closed as integrated Apache Pulsar into TiCDC by pingcap/tiflow#751 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Incubating Program
Add Apache Pulsar to
tidb-binlog.Describe the feature or project you want to incubate:
Summary
tidb-binlogis a very nice and efficient Change Data Capture(CDC) tool, introducing a new component Apache Pulsar to further improve the processing power oftidb-binlog.Motivation
During the Change Data Capture(CDC), we need to ensure the order of the messages. Aka, we need to ensure that DDL arrives before DML. Currently, we use Kafka to process the order of messages, but as you know, Kafka can only guarantee the order of messages within a single partition. If we need to expand the downstream data processing capabilities, sometimes we want more partitions to provide more processing power, but how do we ensure the order of messages at this time? This is a trickier question.
Here, please allow me to introduce a new feature of Apache Pulsar: key_shared It ensures that messages are distributed to the key in order different partitions, this sounds like a very nice feature. Can help
tidb-binlogto improve processing performance, at the same time, Pulsar has other great features, please refer to Apache Pulsar.This is one of the reasons I introduce Apache Pulsar, and the other is we can consider making
tidb-binlogsupport pluggable interfaces while allowing support for Kafka, Pulsar and other message queues, which can give users more choices.Estimated Time
30 days
The text was updated successfully, but these errors were encountered: