TiDB Cluster supports snapshot level consistency replication in time

[amyangfei]

Feature Request

Is your feature request related to a problem? Please describe:

In TiCDC we provide multiple level consistency replication, including row-level ordered, table level ordered, and eventual transaction consistency. Users also pay attention to the transaction guarantee, for example

• Can TiCDC guarantee the atomic replication of a transaction, which interacts with multiple tables?
• Can TiCDC guarantee the atomic replication of a single table transaction?

In the current implementation of MySQL sink, multiple table transaction can't be replicated atomically, but table-level transaction atomic is supported.

What's more, in many scenarios users want to ensure the downstream is replicated to a globally consistent state, as TiCDC supports eventual transaction consistency, and TiDB supports snapshot read, we could combine these two features and provide a snapshot level consistency replication strategy.

TODO list

• Store a safepoint mapping in downstream, recording the TSO mapping between upstream and downstream. Each two matched TSOs guarantee the same snapshot between upstream and downstream.
• Make MySQL sink supports to replicate to a globally consistent state dynamically. (Dynamic means during the replication of TiCDC, we can acquire globally consistent state continuously)
• Besides we should consider the possible usage scenarios of this feature.

Note this strategy only supports replicating from TiDB to TiDB.

Describe alternatives you've considered:

Currently the MySQL sink only replicates to an eventually consistent state if and only if we provide a target-ts in the changefeed, this is caused by the multiple captures and concurrent execution in MySQL sink.

Teachability, Documentation, Adoption, Migration Strategy:

Value

Value description

This feature enables TiCDC to replicate to TiDB/MySQL with eventual transaction consistency

Value score

• (TBD) / 5

Workload estimation

• (TBD) person-day

Time

GanttStart: 2020-09-01
GanttDue: 2020-10-31

[amyangfei]

Task closed, the overall usage scenario will be completed in #691