Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support snapshot level consistent replication #691

Closed
1 of 3 tasks
amyangfei opened this issue Jun 24, 2020 · 4 comments
Closed
1 of 3 tasks

Support snapshot level consistent replication #691

amyangfei opened this issue Jun 24, 2020 · 4 comments
Assignees
@amyangfei
Labels
difficulty/hard Hard task. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/P0 The issue has P0 priority. subject/new-feature Denotes an issue or pull request adding a new feature.
Milestone
v5.1.4

Comments

@amyangfei
Copy link
Contributor

amyangfei commented Jun 24, 2020 •
edited
Loading

Feature Request

Is your feature request related to a problem? Please describe:

In many scenarios, users use TiCDC to backup data from cluster A to cluster B, and use cluster B as a plan-B if cluster A meets unrecoverable disaster.

Currently TiCDC has several drawbacks:

  • If upstream PD cluster meets disaster, TiCDC can't replicate until upstream PD recovers. Besides as TiCDC has multiple captures and even in on processor DML execution is concurrent, so in this case, the downstream can't be mapped to a snapshot against upstream.
  • If upstream TiKV meets disaster, some received data from upstream can't be replicated to downstream due to some resolved ts missing from regions.
  • MySQL sink in TiCDC can't ensure downstream can be mapped to a snapshot against upstream in real time, this is also the main point focused in TiDB Cluster supports snapshot level consistency replication in time #658

Note: This work will be a part of PITR

Related docs: Support read_only Global Variable in TiDB

Task List

  • Support MySQL sink can replicate to a snapshot level consistent state when upstream meets a disaster

  • Support log sink can replicate to a snapshot level consistent state even when upstream meets a disaster

  • pingcap/ticdc#658 Snapshot Level Consistency Replication in non-disaster scenario

Progress Tracking

Describe alternatives you've considered:

Have some possible solutions:

  • use PD to coordinate to a consistent TSO, if upstream PD cluster is available
  • add undo log mechanism, use it to recover downstream to a consistent TSO if upstream PD cluster is not available.

Value

Value description

(TBD)

Value score

  • (TBD) / 5

Workload estimation

  • 40 person-day

Time

GanttStart: 2020-09-01
GanttDue: 2020-10-31
GanttProgress: 20%
@amyangfei amyangfei added priority/P0 The issue has P0 priority. status/need-discussion Issue that needs to be discussed to confirm priority, milestone, plan and task breakdown. labels Jun 24, 2020
@amyangfei amyangfei added this to the v5.0.0 milestone Jun 24, 2020
@amyangfei amyangfei added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. and removed status/need-discussion Issue that needs to be discussed to confirm priority, milestone, plan and task breakdown. labels Jun 24, 2020
@amyangfei
Copy link
Contributor Author

ref: pingcap/br#325

@scsldb scsldb added the subject/new-feature Denotes an issue or pull request adding a new feature. label Jul 3, 2020
@scsldb scsldb modified the milestones: v5.0.0-alpha, v5.0.0-beta.1 Jul 15, 2020
@IANTHEREAL IANTHEREAL mentioned this issue Jul 16, 2020
Open
@IANTHEREAL
Copy link
Collaborator

one solution - pingcap/tidb#18426 (comment)

@IANTHEREAL IANTHEREAL changed the title Support snapshot level consistent replication in disaster Support snapshot level consistent replication Jul 28, 2020
@kennytm kennytm mentioned this issue Jul 28, 2020
Open
12 tasks
@amyangfei amyangfei added the difficulty/hard Hard task. label Jul 29, 2020
@IANTHEREAL
Copy link
Collaborator

@amyangfei can you update this issue to current status, contain our newest plan

This was referenced Nov 4, 2020
Closed
Closed
@amyangfei amyangfei modified the milestones: v5.0.0-beta.1, v5.0.0 Mar 2, 2021
@amyangfei amyangfei modified the milestones: v5.0.0, v5.0.1 Apr 9, 2021
@amyangfei amyangfei modified the milestones: v5.0.1, v5.0.2 Apr 26, 2021
@lonng lonng modified the milestones: v5.0.2, v5.0.3 Jun 2, 2021
@zier-one zier-one modified the milestones: v5.0.3, v5.1.1 Jun 25, 2021
@amyangfei amyangfei modified the milestones: v5.1.1, v5.1.2 Jul 30, 2021
@amyangfei amyangfei modified the milestones: v5.1.2, v5.1.3 Sep 28, 2021
@amyangfei amyangfei modified the milestones: v5.1.3, v5.1.4 Dec 8, 2021
@amyangfei
Copy link
Contributor Author

This feature has been implemented in #2351

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amyangfei amyangfei

Labels
difficulty/hard Hard task. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/P0 The issue has P0 priority. subject/new-feature Denotes an issue or pull request adding a new feature.
Projects
None yet
Milestone
v5.1.4
Development

No branches or pull requests
5 participants
@amyangfei @lonng @scsldb @IANTHEREAL @zier-one