Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

schema_storage: fix schema GC threshold & improve memory management #3172

Merged
merged 34 commits into from
Nov 4, 2021
Merged

schema_storage: fix schema GC threshold & improve memory management #3172

merged 34 commits into from
Nov 4, 2021

Conversation

liuzix
Copy link
Contributor

@liuzix liuzix commented Oct 29, 2021 •
edited
Loading

What problem does this PR solve?

What is changed and how it works?

  • Correct the calculation of the threshold used to GC schemas.
  • Optimize the slice operations used when GC'ing the schemas.

Check List

Tests

  • Unit test
  • Integration test

Related changes

  • Need to cherry-pick to the release branch

Release note 

Fix memory leak after processing DDLs.

@liuzix liuzix added type/bugfix This PR fixes a bug. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Oct 29, 2021
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Oct 29, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@liuzix liuzix added this to the v5.3.0 milestone Oct 29, 2021
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Oct 29, 2021
@ti-chi-bot ti-chi-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 29, 2021
@overvenus overvenus added needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. labels Oct 29, 2021
@liuzix liuzix added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 29, 2021
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 29, 2021
@liuzix liuzix added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 29, 2021
@liuzix liuzix changed the title schema_storage: make runtime GC snapshots in time [WIP] schema_storage: make runtime GC snapshots in time Oct 29, 2021
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 29, 2021
@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 29, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Oct 29, 2021

/run-integration-tests

@codecov-commenter
Copy link

codecov-commenter commented Oct 29, 2021 •
edited
Loading

Codecov Report

Merging #3172 (0092257) into master (de2fca8) will increase coverage by 0.2075%.
The diff coverage is 88.8888%.

@@               Coverage Diff                @@
##             master      #3172        +/-   ##
================================================
+ Coverage   56.7226%   56.9302%   +0.2075%     
================================================
  Files           214        214                
  Lines         22915      22914         -1     
================================================
+ Hits          12998      13045        +47     
+ Misses         8604       8536        -68     
- Partials       1313       1333        +20

@liuzix
Copy link
Contributor Author

liuzix commented Oct 29, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Oct 29, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Nov 4, 2021

/run-kafka-tests

@liuzix
Copy link
Contributor Author

liuzix commented Nov 4, 2021

/run-kakfa-integration-test

@Rustin170506
Copy link
Member

/run-kafka-integration-test

@Rustin170506 Rustin170506 removed the status/ptal Could you please take a look? label Nov 4, 2021
@ti-chi-bot ti-chi-bot merged commit fd39bb2 into pingcap:master Nov 4, 2021
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 4, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#3172
644c08f 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 4, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3273.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 4, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#3172
0bc93ab 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 4, 2021
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3274.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 4, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#3172
a241934 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 4, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3275.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 4, 2021
@liuzix @ti-chi-bot
This is an automated cherry-pick of pingcap#3172
78751f5 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 4, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3276.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: cannot checkout 5.3: error checking out 5.3: exit status 1. output: error: pathspec '5.3' did not match any file(s) known to git

@overvenus
Copy link
Member

/run-cherry-picker

@overvenus overvenus added needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. and removed needs-cherry-pick-5.3 labels Nov 4, 2021
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 4, 2021
@liuzix @ti-chi-bot
schema_storage: fix schema GC threshold & improve memory management (p…
821d3bc 
…ingcap#3172)
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 4, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3280.

overvenus pushed a commit that referenced this pull request Nov 5, 2021
@ti-chi-bot @liuzix
schema_storage: fix schema GC threshold & improve memory management (#…
21cdf26 
…3172) (#3280)

Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
ti-chi-bot added a commit that referenced this pull request Nov 12, 2021
@ti-chi-bot
schema_storage: fix schema GC threshold & improve memory management (#…
ce42013 
…3172) (#3273)
ti-chi-bot added a commit that referenced this pull request Dec 15, 2021
@ti-chi-bot
schema_storage: fix schema GC threshold & improve memory management (#…
466e978 
…3172) (#3275)
liuzix added a commit to ti-chi-bot/tiflow that referenced this pull request Jan 27, 2022
@liuzix
This is an automated cherry-pick of pingcap#3172
16289fe 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot added a commit that referenced this pull request Jan 27, 2022
@ti-chi-bot
schema_storage: fix schema GC threshold & improve memory management (#…
3883f52 
…3172) (#3276)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei approved these changes

@overvenus overvenus overvenus approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
v5.3.0
Development

Successfully merging this pull request may close these issues.

schema storage consumes too much memory after schemas should have been GC'd
6 participants
@liuzix @ti-chi-bot @codecov-commenter @overvenus @Rustin170506 @amyangfei