Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

puller: fix a memory leak bug in the frontier #704

Merged
merged 5 commits into from
Jul 3, 2020

Conversation

zier-one
Copy link
Contributor

@zier-one zier-one commented Jun 29, 2020

What problem does this PR solve?

fix a memory leak bug in the frontier

after this pr, nodes in the frontier could only be added, not removed, which led to the memory leak issue

In this pr, when regions are merged, the frontier will remove redundant nodes and keeps only the necessary nodes

fix #683

What is changed and how it works?

Benchmark

Before this PR

goos: darwin
goarch: amd64
pkg: github.com/pingcap/ticdc/cdc/puller/frontier
BenchmarkSpanFrontier/5k-8           6533013           183 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontier/10k-8          5940601           206 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontier/50k-8          4927638           252 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontier/100k-8         4529256           242 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/5k_5-8              5396868           213 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/5k_10-8             4033922           300 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/5k_100-8            1000000          1169 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/5k_500-8             251289          4662 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/10k_5-8             4996110           230 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/10k_10-8            4038318           309 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/10k_100-8           1000000          1123 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/10k_500-8            251916          5005 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/50k_5-8             4139940           319 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/50k_10-8            3498513           364 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/50k_100-8           1000000          1143 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/50k_500-8            248053          4866 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/100k_5-8            3539360           316 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/100k_10-8           3341758           366 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/100k_100-8          1000000          1225 ns/op           0 B/op          0 allocs/op
BenchmarkSpanFrontierOverlap/100k_500-8           248091          5223 ns/op           0 B/op          0 allocs/op

After This PR

goos: darwin
goarch: amd64
pkg: github.com/pingcap/ticdc/cdc/puller/frontier
BenchmarkSpanFrontier/5k-8       	 4541343	       281 ns/op	      96 B/op	       1 allocs/op
BenchmarkSpanFrontier/10k-8      	 4640968	       261 ns/op	      96 B/op	       1 allocs/op
BenchmarkSpanFrontier/50k-8      	 3709297	       319 ns/op	      96 B/op	       1 allocs/op
BenchmarkSpanFrontier/100k-8     	 3390427	       337 ns/op	      96 B/op	       1 allocs/op
BenchmarkSpanFrontierOverlap/5k_5-8         	 1335870	       796 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/5k_10-8        	 1721634	       697 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/5k_100-8       	 1721421	       698 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/5k_500-8       	 1768676	       684 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/10k_5-8        	 1570315	       773 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/10k_10-8       	 1536466	       800 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/10k_100-8      	 1515050	       838 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/10k_500-8      	 1584950	       797 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/50k_5-8        	 1426015	       877 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/50k_10-8       	 1377284	       903 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/50k_100-8      	 1381473	       834 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/50k_500-8      	 1408707	       873 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/100k_5-8       	 1346317	       913 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/100k_10-8      	 1337869	       896 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/100k_100-8     	 1356310	       948 ns/op	     374 B/op	       6 allocs/op
BenchmarkSpanFrontierOverlap/100k_500-8     	 1435084	       841 ns/op	     374 B/op	       6 allocs/op

Check List

Tests

  • Unit test
  • Integration test

Release note

@zier-one
Copy link
Contributor Author

/run-all-tests

@zier-one
Copy link
Contributor Author

/run-integration-tests

@codecov-commenter
Copy link

codecov-commenter commented Jun 30, 2020

Codecov Report

Merging #704 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #704   +/-   ##
===========================================
  Coverage   32.6823%   32.6823%           
===========================================
  Files            90         90           
  Lines          9473       9473           
===========================================
  Hits           3096       3096           
  Misses         6132       6132           
  Partials        245        245           

@amyangfei amyangfei added type/bugfix This PR fixes a bug. status/ptal Could you please take a look? labels Jul 1, 2020
@amyangfei amyangfei added this to the v4.0.3 milestone Jul 1, 2020
cdc/puller/frontier/frontier_bench_test.go Show resolved Hide resolved
cdc/puller/frontier/list.go Outdated Show resolved Hide resolved
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amyangfei amyangfei added the LGT1 label Jul 3, 2020
@amyangfei
Copy link
Contributor

/run-integration-tests

@zier-one
Copy link
Contributor Author

zier-one commented Jul 3, 2020

/run-integration-tests

@zier-one zier-one added LGT2 and removed LGT1 status/ptal Could you please take a look? labels Jul 3, 2020
@zier-one zier-one merged commit 39bb362 into pingcap:master Jul 3, 2020
@zier-one zier-one deleted the fix_frontier_leak branch July 3, 2020 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

There is a memory leak in the frontier
4 participants