Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syncer(dm): fix different output format for operate-schema get #5824

Merged
merged 6 commits into from
Jun 15, 2022
Merged

syncer(dm): fix different output format for operate-schema get #5824

merged 6 commits into from
Jun 15, 2022

Conversation

ForwardStar
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #5688

What is changed and how it works?

  • If the checkpoint doesn't exist and the schema is retrieved from the downstream, replace the downstream table name with the requested upstream table name

  • Delete all \n in the SQL results for the operate-schema result

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility? No.
Do you need to update user documentation, design documentation or monitoring documentation? No.

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jun 10, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. release-note-none Denotes a PR that doesn't merit a release note. labels Jun 10, 2022
@sre-bot
Copy link

sre-bot commented Jun 10, 2022

CLA assistant check
All committers have signed the CLA.

@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 10, 2022
@ForwardStar
Copy link
Contributor Author

/cc @dsdashun @lance6716

@ForwardStar
Copy link
Contributor Author

/run-leak-test

@codecov-commenter
Copy link

codecov-commenter commented Jun 10, 2022

Codecov Report

Merging #5824 (eafcd74) into master (fcea4d5) will increase coverage by 0.6140%.
The diff coverage is 57.4834%.

Flag Coverage Δ
cdc 61.9898% <62.6707%> (+0.8417%) ⬆️
dm 52.0934% <21.8750%> (+0.0507%) ⬆️
engine 58.7230% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5824        +/-   ##
================================================
+ Coverage   56.0768%   56.6908%   +0.6140%     
================================================
  Files           535        676       +141     
  Lines         70143      79018      +8875     
================================================
+ Hits          39334      44796      +5462     
- Misses        27078      30022      +2944     
- Partials       3731       4200       +469     

@lance6716 lance6716 changed the title dm/syncer: fix different output format for operate-schema get syncer(dm): fix different output format for operate-schema get Jun 10, 2022
@lance6716 lance6716 added the area/dm Issues or PRs related to DM. label Jun 10, 2022
@@ -76,7 +77,9 @@ func (s *Syncer) OperateSchema(ctx context.Context, req *pb.OperateWorkerSchemaR
s.tctx.L().Info("table schema is not in checkpoint, fetch from downstream",
zap.String("table", sourceTable.String()))
targetTable := s.route(sourceTable)
return dbconn.GetTableCreateSQL(s.tctx.WithContext(ctx), s.downstreamTrackConn, targetTable.String())
result, err2 := dbconn.GetTableCreateSQL(s.tctx.WithContext(ctx), s.downstreamTrackConn, targetTable.String())
result = strings.ReplaceAll(result, fmt.Sprintf("CREATE TABLE `%s`", targetTable.Name), fmt.Sprintf("CREATE TABLE `%s`", sourceTable.Name))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can replace the first occurrence, not all occurrence., in case user has writen the "CREATE TABLE ``" pattern in column comments

And use quotes.QuoteName to correctly escape the ` in table name.

syncer.checkpoint.(*RemoteCheckPoint).points = make(map[string]map[string]*binlogPoint)

mock.ExpectQuery("SHOW CREATE TABLE " + "`test_1`.`t_2`").WillReturnRows(
sqlmock.NewRows([]string{"Table", "Create Table"}).
Copy link
Contributor

@lance6716 lance6716 Jun 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this mock is not the same output format as a real MySQL/TiDB. We should make sure the test is valid.

@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.0 Should cherry pick this PR to release-6.0 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. labels Jun 13, 2022
dm/syncer/syncer_test.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 15, 2022
Copy link
Contributor

@dsdashun dsdashun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member

@dsdashun: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@lance6716 lance6716 added the require-LGT1 Indicates that the PR requires an LGTM. label Jun 15, 2022
@lance6716
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: eafcd74

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 15, 2022
@ti-chi-bot ti-chi-bot merged commit 1ba1471 into pingcap:master Jun 15, 2022
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 15, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5884.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 15, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5885.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5886.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 15, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5887.

@ForwardStar
Copy link
Contributor Author

/cherry-pick-release-5.4

@ForwardStar
Copy link
Contributor Author

/cherry-pick release-5.4

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 24, 2022
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

@ForwardStar: new pull request created: #6031.

In response to this:

/cherry-pick release-5.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dm Issues or PRs related to DM. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.0 Should cherry pick this PR to release-6.0 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. release-note-none Denotes a PR that doesn't merit a release note. require-LGT1 Indicates that the PR requires an LGTM. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

different output format for operate-schema get
6 participants