TiCDC supports a snapshot-map in replication #932

Colins110 · 2020-09-07T12:54:31Z

What problem does this PR solve?

TiDB Cluster supports snapshot level consistency replication in time

What is changed and how it works?

Make MySQL sink supports to replicate to a global consistent state dynamically.
set a syncpoint at regular intervals for get globally consistent state;
by stopping updating the global resolvedTS util the global checkpointTS == global resolvedTS, then set the sync point TS
=checkpointTS;
Store a safepoint mapping in downstream, recording the TSO mapping between upstream and downstream.
the map in the downstream like this:

cf	primary_ts	secondary_ts
sample	419288824469258241	419288824566251551
sample	419288981663907844	419288981957509121

Check List

Tests

Manual test
Integration Test
a integration test named syncpoint test the syncpoint's correctness by sync_diff_inspector tool. you can use make integration_test CASE=syncpoint to run the test.

Code changes

in cdc/changeFeed.go:
add function:
startSyncPeriod()
createSynctable()
sinkSyncpoint(ctx context.Context)

add some property in changeFeed struct

some change in function calcResolvedTs(ctx context.Context):
to judge the syncpoint and sink the map to downstream

in cdc/owner.go:
add function:
configureSinkURI(ctx context.Context, dsnCfg *dmysql.Config, tz *time.Location)
createSyncpointSink(ctx context.Context, info *model.ChangeFeedInfo, id string)

some change in function newChangeFeed( ctx context.Context, id model.ChangeFeedID, processorsInfos model.ProcessorsInfos, taskPositions map[string]*model.TaskPosition, info *model.ChangeFeedInfo, checkpointTs uint64):
for every changefeed,we need create a standalone mysql link to record the syncpoint map in the future.

the new flags for cdc cli changefeed:
you can open the feature to record the syncpoint like this when you create a new changefeed or update a changefeed:

set on create
cdc changefeed create --sink-uri="$SINK_URI" --sync-point --sync-interval=10s
set on update

cdc cli changefeed pause -c test-cf
cdc cli changefeed update -c test-cf --sink-uri="mysql://127.0.0.1:3306/" --sync-point --sync-interval=10m
cdc cli changefeed resume -c test-cf

the --sync-point mean open the syncpoint feature, the --sync-interval can set the interval for recording the sync point(the default value is 10min).

Release note

Support snapshot level consistency replication in time

…point has not add yet)

CLAassistant · 2020-09-07T12:54:37Z

All committers have signed the CLA.

cdc/changefeed.go

Dockerfile

cdc/changefeed.go

cdc/owner.go

cdc/changefeed.go

liuzix · 2020-09-08T16:15:04Z

@Colins110 Could you explain more about how a user would make use of this feature? What is the potential user scenario and how would the user make use of the "safepoint mapping"?

…eate'

Colins110 · 2020-09-14T03:01:50Z

@Colins110 Could you explain more about how a user would make use of this feature? What is the potential user scenario and how would the user make use of the "safepoint mapping"?

you can refer to this issue: TiDB Cluster supports snapshot level consistency replication in time

cdc/owner.go

cdc/changefeed.go

cdc/owner.go

cmd/client_changefeed.go

tests/syncpoint/run.sh

README.md

cdc/changefeed.go

amyangfei · 2020-09-15T10:31:02Z

/run-integration-tests tidb=release-4.0 tikv=release-4.0 pd=release-4.0

cdc/changefeed.go

codecov-io · 2020-10-09T07:00:41Z

Codecov Report

Merging #932 into master will decrease coverage by 0.6336%.
The diff coverage is 1.9607%.

@@               Coverage Diff                @@
##             master       #932        +/-   ##
================================================
- Coverage   32.1982%   31.5646%   -0.6337%     
================================================
  Files           100        101         +1     
  Lines         10836      11025       +189     
================================================
- Hits           3489       3480         -9     
- Misses         6956       7152       +196     
- Partials        391        393         +2

cdc/owner.go

cdc/sink/mysql.go

ti-srebot · 2020-10-10T01:06:49Z

@Colins110,Thanks for your review. The bot only counts LGTMs from Reviewers and higher roles, but you're still welcome to leave your comments.See the corresponding SIG page for more information. Related SIG: migrate(slack).

zier-one · 2020-10-10T09:19:18Z

cdc/owner.go

@@ -333,13 +333,13 @@ func (o *Owner) newChangeFeed(
 	}
 	errCh := make(chan error, 1)

-	sink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)
+	primarySink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)


Suggested change

primarySink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)

sink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)

I should use package sink in L360 (var syncpointStore sink.SyncpointStore) to create a syncpointStore,if the var named sink,the package sink will not be used in the follow code,there is a conflict.

Maybe move the sink.SyncpointStore to another package is better, but it reuses some code in the sink package.
We can refine this in another PR.

cdc/sink/mysql.go

amyangfei · 2020-10-12T02:46:25Z

/run-integration-tests

amyangfei

LGTM

zier-one · 2020-10-12T09:04:35Z

LGTM

amyangfei · 2020-10-13T05:24:50Z

/merge

ti-srebot · 2020-10-13T05:27:23Z

/run-all-tests

ti-srebot · 2020-10-13T05:41:22Z

@Colins110 merge failed.

amyangfei · 2020-10-13T05:42:28Z

/run-integration-tests

amyangfei · 2020-10-13T07:15:51Z

/run-integration-tests

amyangfei · 2020-10-13T08:35:21Z

/run-integration-tests

Colins110 added 8 commits August 13, 2020 00:40

create dev branch

4e8689e

add create sync point logic

88318c3

add sync table logic

e4cadbd

add proxy for docker

f26eead

[9.1] fix the logic

3fd1b3d

record the sync point base on timer(the ddl and table migration sync …

97a1615

…point has not add yet)

add ddl syncpoint record

94ec1d4

a liitle change for createSyncpointSink()

6abad54

amyangfei reviewed Sep 7, 2020

View reviewed changes

amyangfei added this to the v5.0.0-alpha milestone Sep 8, 2020

amyangfei added component/replica-model Replication model component. component/sink Sink component. labels Sep 8, 2020

Colins110 added 4 commits September 9, 2020 01:26

9.9 fix for syncpointsinl logic

c13ccc3

add integration test for sync point record

abd124b

9.13 add sync-point and sync-interval flag for 'cdc cli changefeed cr…

091927b

…eate'

add some flags in 'cdc cli changefeed'

0a83609

Colins110 closed this Sep 13, 2020

Colins110 reopened this Sep 13, 2020

fix the changefeed resume logic

1da2a73

Merge branch 'master' into dev

8f25e2f

amyangfei reviewed Sep 15, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

cdc/changefeed.go Outdated Show resolved Hide resolved

Colins110 added 3 commits September 15, 2020 14:49

merge master and fix some err

b1dad1d

fix some err about sql for ci-lint

7e2ca72

fix code fmt

22fca2a

amyangfei reviewed Sep 15, 2020

View reviewed changes

cdc/changefeed.go Outdated Show resolved Hide resolved

cdc/changefeed.go Outdated Show resolved Hide resolved

cdc/changefeed.go Outdated Show resolved Hide resolved

cdc/changefeed.go Outdated Show resolved Hide resolved

zier-one reviewed Oct 9, 2020

View reviewed changes

cdc/owner.go Outdated Show resolved Hide resolved

cdc/owner.go Outdated Show resolved Hide resolved

cdc/sink/mysql.go Outdated Show resolved Hide resolved

cdc/sink/mysql.go Outdated Show resolved Hide resolved

amyangfei reviewed Oct 9, 2020

View reviewed changes

cdc/sink/mysql.go Outdated Show resolved Hide resolved

change some info in log(owner.go)

f6f3993

Colins110 added 2 commits October 10, 2020 16:23

set syncpoint's tablename and schema name to be constants

6ab63c7

change name 'syncpointSink' to 'syncpointLink'

416baee

zier-one reviewed Oct 10, 2020

View reviewed changes

change name 'syncpointLink' to 'syncpointStore'

5045d09

amyangfei reviewed Oct 12, 2020

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 12, 2020

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Oct 12, 2020

ti-srebot approved these changes Oct 12, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Oct 12, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 13, 2020

Merge branch 'master' into dev

3383eab

amyangfei merged commit 3c08cc7 into pingcap:master Oct 13, 2020

amyangfei mentioned this pull request Oct 15, 2020

support flashback table to any point in time pingcap/tidb#20302

Open

IANTHEREAL changed the title ~~TiDB Cluster supports snapshot level consistency replication in time~~ TiCDC supports a snapshot-map in replication Oct 28, 2020

amyangfei mentioned this pull request Jul 22, 2021

Subtask tracing issue for undo based eventually consistent replication #2352

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TiCDC supports a snapshot-map in replication #932

TiCDC supports a snapshot-map in replication #932

Colins110 commented Sep 7, 2020 •

edited

Loading

CLAassistant commented Sep 7, 2020 •

edited

Loading

liuzix commented Sep 8, 2020

Colins110 commented Sep 14, 2020

amyangfei commented Sep 15, 2020

codecov-io commented Oct 9, 2020

ti-srebot commented Oct 10, 2020

zier-one Oct 10, 2020

Colins110 Oct 10, 2020

amyangfei Oct 12, 2020

amyangfei commented Oct 12, 2020

amyangfei left a comment

zier-one commented Oct 12, 2020

amyangfei commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

amyangfei commented Oct 13, 2020

amyangfei commented Oct 13, 2020

amyangfei commented Oct 13, 2020

	primarySink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)
	sink, err := sink.NewSink(ctx, id, info.SinkURI, filter, info.Config, info.Opts, errCh)

TiCDC supports a snapshot-map in replication #932

TiCDC supports a snapshot-map in replication #932

Conversation

Colins110 commented Sep 7, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CLAassistant commented Sep 7, 2020 • edited Loading

liuzix commented Sep 8, 2020

Colins110 commented Sep 14, 2020

amyangfei commented Sep 15, 2020

codecov-io commented Oct 9, 2020

Codecov Report

ti-srebot commented Oct 10, 2020

zier-one Oct 10, 2020

Choose a reason for hiding this comment

Colins110 Oct 10, 2020

Choose a reason for hiding this comment

amyangfei Oct 12, 2020

Choose a reason for hiding this comment

amyangfei commented Oct 12, 2020

amyangfei left a comment

Choose a reason for hiding this comment

zier-one commented Oct 12, 2020

amyangfei commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

amyangfei commented Oct 13, 2020

amyangfei commented Oct 13, 2020

amyangfei commented Oct 13, 2020

Colins110 commented Sep 7, 2020 •

edited

Loading

CLAassistant commented Sep 7, 2020 •

edited

Loading