Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mounter(ticdc): fix decoding when upstream doesn't enable new collation #11370

Merged
merged 12 commits into from
Jul 2, 2024

Conversation

lidezhu
Copy link
Collaborator

@lidezhu lidezhu commented Jul 2, 2024

What problem does this PR solve?

Issue Number: close #11371

What is changed and how it works?

Explicitly call collate.SetNewCollationEnabledForTest(false) before run cdc.

Previously, this function was called in a tidb package. But it was removed as described in pingcap/tidb#52191 (comment).

If the upstream doesn't enable new collation, the primary key is not encoded in the value part when the table is a cluster index table. And we rely on the following line to decode the primary key.

datums, err = tablecodec.DecodeHandleToDatumMap(

But this line only decode the primary key when the global variable newCollationEnabled in tidb package is false. (It means collate.SetNewCollationEnabledForTest(false) must be called.)

Check List

Tests

  • Integration test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix the issue that TiCDC cannot correctly decode primary key of table with cluster index when the upstream does not enable new collation.

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 2, 2024
@lidezhu lidezhu changed the title fix collation and add tests mounter(ticdc): fix decoding when upstream doesn't enable new collation Jul 2, 2024
Copy link

codecov bot commented Jul 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.6033%. Comparing base (7ddb1a3) to head (715e8e4).

Additional details and impacted files
Components Coverage Δ
cdc 61.3676% <ø> (+0.0127%) ⬆️
dm 51.2071% <ø> (+0.0100%) ⬆️
engine 63.4020% <ø> (+0.0282%) ⬆️
Flag Coverage Δ
unit 57.6033% <ø> (+0.0134%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #11370        +/-   ##
================================================
+ Coverage   57.5899%   57.6033%   +0.0134%     
================================================
  Files           849        849                
  Lines        126286     126286                
================================================
+ Hits          72728      72745        +17     
+ Misses        48144      48128        -16     
+ Partials       5414       5413         -1     

@lidezhu
Copy link
Collaborator Author

lidezhu commented Jul 2, 2024

/retest

cmd/cdc/main.go Outdated
"github.com/pingcap/tiflow/pkg/cmd"
)

func main() {
// NOTE: the line is removed from TiDB repo in https://github.com/pingcap/tidb/pull/52191#issuecomment-2024836481.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add more comment to explain why we need to call this function.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

@ti-chi-bot ti-chi-bot added the needs-cherry-pick-release-8.2 Should cherry pick this PR to release-8.2 branch. label Jul 2, 2024
@lidezhu
Copy link
Collaborator Author

lidezhu commented Jul 2, 2024

/test verify

@ti-chi-bot ti-chi-bot added the needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. label Jul 2, 2024
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 2, 2024
@ti-chi-bot ti-chi-bot bot added the approved label Jul 2, 2024
Copy link
Contributor

ti-chi-bot bot commented Jul 2, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, CharlesCheung96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [CharlesCheung96,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 2, 2024
Copy link
Contributor

ti-chi-bot bot commented Jul 2, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-07-02 10:56:19.669055013 +0000 UTC m=+1321906.154543845: ☑️ agreed by CharlesCheung96.
  • 2024-07-02 14:06:05.852061112 +0000 UTC m=+1333292.337549941: ☑️ agreed by asddongmen.

@ti-chi-bot ti-chi-bot bot merged commit da9a6ac into pingcap:master Jul 2, 2024
27 of 28 checks passed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.1: #11378.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.2: #11379.

@lidezhu lidezhu deleted the fix-collate branch July 2, 2024 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.2 Should cherry pick this PR to release-8.2 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TiCDC meet duplicate entry error when run ycsb insert workload
5 participants