sink(ticdc): remove force consume when redo is disabled

[CharlesCheung96]

What problem does this PR solve?

Issue Number: close #5453, close #1683

What is changed and how it works?

In order to solve the OOM problem of single table, we refactor flowControl and Sink modules on the basis of splitting transaction, so that they can obtain data in batches and send it to the downstream, which reduces the memory consumption of individual table to constant level:

1. In flowcontrol module, allocate and release based on resolvedTs and batchID.
2. In sink module, replace model.Ts with model.ResolvedTs to support batch-based memory release.
3. In sinkNode, release memory based on checkpointTs and batchID.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

down stream: TiDB
total row of txn: 30 million
txn size: ~5.4 GB
downstream: tidb cluster
memory of cdc node: 32 GB

The results show that memory consumption of single table is constant. However, a lower quota value may affect throughput, so it is recommended to increase the quota of single tables to 30-40 MB.

compare with #5454
quota: 100MB
down stream: TiDB
total row of txn: 30 million
txn size: ~5.4 GB
downstream: tidb cluster
memory of cdc node: 32 GB

up-TiCDC_2022-06-06T10_33_19.155Z.json.zip

heap_profile.zip

compare with #5454
quota: 35MB
down stream: Kafka
total row of txn: 30 million
txn size: ~5.4 GB
downstream: tidb cluster
memory of cdc node: 32 GB
Test-Cluster-TiCDC-1-12_2022-06-09T11_16_25.947Z.json.zip

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

`Optimize the memory consumption required for TiCDC to process large transactions`.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• amyangfei
• asddongmen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[CharlesCheung96]

/run-all-tests

[nongfushanquan]

/cc @hi-rustin @hicqu

[nongfushanquan]

could you add some comments to explain the method to solve the problem

[CharlesCheung96]

could you add some comments to explain the method to solve the problem

Done, PTAL.

[CharlesCheung96]

/hold

[CharlesCheung96]

/run-all-tests

[codecov-commenter]

Codecov Report

Merging #5712 (e4ba9d5) into master (b9773ac) will increase coverage by 0.0982%.
The diff coverage is 62.1621%.

❗ Current head e4ba9d5 differs from pull request most recent head 4ef0e31. Consider uploading reports for the commit 4ef0e31 to get more accurate results

Flag	Coverage Δ
cdc	62.1435% <66.6666%> (+0.2122%)	⬆️
dm	51.9454% <52.0833%> (+0.0265%)	⬆️
engine	61.6326% <70.5882%> (+0.0394%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5712        +/-   ##
================================================
+ Coverage   56.9653%   57.0635%   +0.0982%     
================================================
  Files           681        682         +1     
  Lines         79903      79995        +92     
================================================
+ Hits          45517      45648       +131     
+ Misses        30106      30069        -37     
+ Partials       4280       4278         -2     

[CharlesCheung96]

/unhold

[amyangfei]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 4ef0e31