br: ebs volume snapshot backup and restore with flashback solution #38700

fengou1 · 2022-10-28T02:08:20Z

What problem does this PR solve?

Adapts Flashback solution.

previously, BR delete the key from rocks DB to get transaction level consistency, however, there is a defect when the compaction filter gc started after the restore was done. the data consistency is broken. v6.3.0 take a workaround compaction during the restore, this is too heavy. the flashback solution is to try to rewrite the MVCC key with the latest ts, it is more effective during the restore compared to compaction.

Issue Number: close #36215

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

apply a tc yaml with recoveryMode:true
apply a restore yaml with restoreMode: volume-snapshot
do a admin checksum for all user tables
Admin checksum show values are the same as backup's

No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-10-28T02:08:22Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3pointer
Defined2014

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

fengou1 · 2022-10-28T02:10:02Z

/cc @3pointer

fengou1 · 2022-10-28T03:38:20Z

/run check-issue-triage-complete

br/pkg/restore/data.go

Defined2014 · 2022-11-02T03:01:12Z

br/pkg/restore/data.go

+		return errors.Trace(err)
+	}
+
+	recovery.progress.Inc()


Could get region count by rangetask.TaskStat in the prepare phase and show progress in FlashbackToVersion.

3pointer · 2022-11-02T03:04:52Z

br/pkg/task/restore_data.go


 	// restore tikv data from a snapshot volume
 	var totalRegions int

-	totalRegions, err = restore.RecoverData(ctx, resolveTs, allStores, mgr, progress)
+	totalRegions, err = restore.RecoverData(ctx, resolveTs, allStores, mgr, progress, restoreTS, cfg.Concurrency)


it seems restoreTs > resolveTs. is it ok to stop gc at restoreTs?

keep safe point in restoreTs is safe since GC never work before the TiDB node startup. The code may need a refactoring to remove the safepointkeeper service, however, I would like to remove it after a thorough test.

…and commitTs to commitTS

3pointer

LGTM

ti-chi-bot · 2022-11-03T02:33:41Z

This pull request has been accepted and is ready to merge.

Commit hash: e501d50

fengou1 · 2022-11-03T03:20:44Z

/run-check_dev

fengou1 · 2022-11-03T03:22:12Z

/run-check_dev_2

fengou1 · 2022-11-03T03:54:28Z

/run-check_dev

fengou1 · 2022-11-03T05:56:22Z

/run-mysql-test

YuJuncen · 2022-11-03T07:32:27Z

/merge

ti-chi-bot · 2022-11-03T07:32:31Z

This pull request has been accepted and is ready to merge.

Commit hash: e531b69

sre-bot · 2022-11-03T08:51:22Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-ddl-test	🟥 failed 1, success 5, total 6	25 min	New failing
idc-jenkins-ci-tidb/common-test	🔴 failed 1, success 10, total 11	10 min	Existing failure
idc-jenkins-ci-tidb/sqllogic-test-2	✅ all 28 tests passed	6 min 34 sec	Fixed
idc-jenkins-ci-tidb/sqllogic-test-1	✅ all 26 tests passed	5 min 54 sec	Fixed
idc-jenkins-ci/integration-cdc-test	🟢 all 38 tests passed	19 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 17 tests passed	14 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 24 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 42 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 53 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

fengou1 added 2 commits October 27, 2022 13:53

feat: move ebs resolve data solution to flashback

e15ed35

refactor: add concurrency and progress for flashback solution

b5a140e

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 28, 2022

fengou1 requested a review from Defined2014 October 28, 2022 02:09

ti-chi-bot requested a review from 3pointer October 28, 2022 02:10

fengou1 self-assigned this Oct 28, 2022

Merge branch 'master' into ebs_with_flashback

e44e112

fengou1 removed the do-not-merge/needs-triage-completed label Oct 28, 2022

3pointer reviewed Oct 28, 2022

View reviewed changes

br/pkg/restore/data.go Outdated Show resolved Hide resolved

Defined2014 reviewed Oct 28, 2022

View reviewed changes

br/pkg/restore/data.go Outdated Show resolved Hide resolved

Merge branch 'master' into ebs_with_flashback

4e64ba0

Defined2014 reviewed Oct 28, 2022

View reviewed changes

br/pkg/restore/data.go Outdated Show resolved Hide resolved

fengou1 and others added 5 commits November 1, 2022 23:16

feat: adapts flashback solution with the entire cluster range

ebb79bf

refactor: add ddl flashback concurrency

97f4b7c

Merge branch 'master' into ebs_with_flashback

7ca4452

refactor: remove dependency from flashbackToVersion function

6db3708

fix: make check_dev happy

f0047ca

Defined2014 reviewed Nov 2, 2022

View reviewed changes

3pointer reviewed Nov 2, 2022

View reviewed changes

fengou1 added 2 commits November 2, 2022 14:48

refactor: add progress for flashback, rename resolvedTs to resolveTS …

3df6aeb

…and commitTs to commitTS

fix: remove empty lines to get build ci pass

8a4747b

Defined2014 approved these changes Nov 2, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 2, 2022

3pointer approved these changes Nov 2, 2022

View reviewed changes

ti-chi-bot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 2, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 3, 2022

ti-chi-bot and others added 3 commits November 3, 2022 10:33

Merge branch 'master' into ebs_with_flashback

90adf23

Merge branch 'master' into ebs_with_flashback

0f045e5

Merge branch 'master' into ebs_with_flashback

900d1c8

Merge branch 'master' into ebs_with_flashback

5c0b444

ti-chi-bot added 4 commits November 3, 2022 12:12

Merge branch 'master' into ebs_with_flashback

7b686e1

Merge branch 'master' into ebs_with_flashback

30d904d

Merge branch 'master' into ebs_with_flashback

007a5f4

Merge branch 'master' into ebs_with_flashback

fb74ee1

fengou1 requested a review from a team as a code owner November 3, 2022 06:45

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed status/can-merge Indicates a PR has been approved by a committer. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 3, 2022

fix: update build bazel to get check_dev up

e531b69

fengou1 force-pushed the ebs_with_flashback branch from 9959021 to e531b69 Compare November 3, 2022 07:21

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 3, 2022

fengou1 removed the request for review from a team November 3, 2022 07:25

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 3, 2022

ti-chi-bot added 2 commits November 3, 2022 15:32

Merge branch 'master' into ebs_with_flashback

00347be

Merge branch 'master' into ebs_with_flashback

f5d1fde

ti-chi-bot merged commit bd081f1 into pingcap:master Nov 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br: ebs volume snapshot backup and restore with flashback solution #38700

br: ebs volume snapshot backup and restore with flashback solution #38700

fengou1 commented Oct 28, 2022 •

edited

Loading

ti-chi-bot commented Oct 28, 2022 •

edited

Loading

fengou1 commented Oct 28, 2022

fengou1 commented Oct 28, 2022

Defined2014 Nov 2, 2022

3pointer Nov 2, 2022

fengou1 Nov 2, 2022

3pointer left a comment

ti-chi-bot commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

YuJuncen commented Nov 3, 2022

ti-chi-bot commented Nov 3, 2022

sre-bot commented Nov 3, 2022

br: ebs volume snapshot backup and restore with flashback solution #38700

br: ebs volume snapshot backup and restore with flashback solution #38700

Conversation

fengou1 commented Oct 28, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 28, 2022 • edited Loading

fengou1 commented Oct 28, 2022

fengou1 commented Oct 28, 2022

Defined2014 Nov 2, 2022

Choose a reason for hiding this comment

3pointer Nov 2, 2022

Choose a reason for hiding this comment

fengou1 Nov 2, 2022

Choose a reason for hiding this comment

3pointer left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

fengou1 commented Nov 3, 2022

YuJuncen commented Nov 3, 2022

ti-chi-bot commented Nov 3, 2022

sre-bot commented Nov 3, 2022

TiDB MergeCI notify

fengou1 commented Oct 28, 2022 •

edited

Loading

ti-chi-bot commented Oct 28, 2022 •

edited

Loading