optimization: Unified Sorter #972

liuzix · 2020-09-27T10:02:51Z

What problem does this PR solve?

#945

Improves serialization performance.
Supports concurrent sorting.
Enables automatic fallback to disk when memory is not enough for sorting.

What is changed and how it works?

Implemented UnifiedSorter.
Made UnifiedSorter the default sorter.
Moved mounting after sorting, to ease serialization to temp files.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change

Side effects

Possible performance regression
Increased code complexity

Benchmarks

Method

We generate mock KV-entries with 1KB data in each entry. We use 256 goroutines to generate data in parallel. There is no resolved event until all data has been fed to the sorter.

Results

Data	Time
50GB	7 min
100GB	16 min
200GB	32 min

Machine specifications

CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Memory: 16GB DDR4 2133MHz * 8
Disk: Intel Corporation NVMe Datacenter SSD

Sorter configuration

{
	NumConcurrentWorker:  16,
	ChunkSizeLimit:       1 * 1024 * 1024 * 1024,   // 1GB
	MaxMemoryPressure:    60,
	MaxMemoryConsumption: 16 * 1024 * 1024 * 1024,  // 16GB
}

Release note

No release note

overvenus · 2020-09-27T11:05:47Z

cdc/puller/unified_sorter.go

+		}
+		// Cache is full. Let GC do its job
+	}
+	panic("Unexpected type")


Prefer log.Fatal.

overvenus · 2020-09-27T11:11:18Z

cdc/puller/unified_sorter.go

+	for i := range heapSorters {
+		finalI := i
+		heapSorters[finalI] = newHeapSorter(finalI, s.pool, sorterOutCh)
+		go func() {


Prefer errgroup.Go to manage related goroutines.

overvenus · 2020-09-27T11:17:25Z

cdc/puller/unified_sorter.go

+	fileBufferSize     = 16 * 1024 * 1024
+	heapSizeLimit      = 4 * 1024 * 1024 // 4MB
+	numConcurrentHeaps = 16
+	memoryLimit        = 1024 * 1024 * 1024 // 1GB


Could you set the value base on system memory capacity, eg. min(upperBound, x% * systemMemory).

overvenus · 2020-09-27T11:23:09Z

cdc/puller/unified_sorter.go

+		memoryUseEstimate: 0,
+		fileNameCounter:   0,
+		mu:                sync.Mutex{},
+		cache:             make([]unsafe.Pointer, 256),


Maybe you want a fixed size array, eg, [256]unsafe.Pointer.

overvenus · 2020-09-27T11:24:22Z

cdc/puller/unified_sorter.go

+}
+
+func newFileSorterBackEnd(fileName string, serde serializerDeserializer) (*fileSorterBackEnd, error) {
+	f, err := os.Create(fileName)


Where do you remove the file?

overvenus · 2020-09-27T11:30:41Z

cdc/puller/unified_sorter.go

+			heapSizeBytesEstimate += event.RawKV.ApproximateSize()
+			if heapSizeBytesEstimate >= heapSizeLimit || isResolvedEvent {


Maybe we need to calculate the estimated size before adding an entry to the heap to make sure it does not exceed the limit.

liuzix · 2020-10-13T07:49:22Z

/run-all-tests

liuzix · 2020-10-13T07:59:20Z

/run-all-tests

liuzix · 2020-10-13T08:15:02Z

/run-all-tests

liuzix · 2020-11-24T14:59:49Z

/run-all-tests

amyangfei · 2020-11-25T02:09:57Z

/run-integration-tests

liuzix · 2020-11-25T03:02:50Z

/run-all-tests

amyangfei · 2020-11-25T03:08:11Z

testing_utils/many_sorters_test/many_sorters.go

@@ -0,0 +1,150 @@
+// Copyright 2020 PingCAP, Inc.


Please exclude testing_utils dir from the coverage counter

… zixiong-unified-sorter

liuzix · 2020-11-25T04:18:28Z

/run-all-tests

amyangfei

rest lgtm

Makefile

Co-authored-by: amyangfei <amyangfei@gmail.com>

amyangfei

LGTM

amyangfei · 2020-11-25T07:04:41Z

/merge

ti-srebot · 2020-11-25T07:04:44Z

Your auto merge job has been accepted, waiting for:

1100
1116

ti-srebot · 2020-11-25T08:29:49Z

/run-all-tests

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-11-25T10:00:38Z

cherry pick to release-4.0 in PR #1122

* cherry pick #972 to release-4.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>

liuzix and others added 11 commits September 18, 2020 14:10

initial commit

e7987be

sync

5b4d676

sync

94142c4

sync

caab34a

sync

55ceabb

sync

0c72c88

sync

a0e99cc

sync

29bfd04

sync

c1505c5

sync

bcd1713

sync

ee55414

liuzix added status/WIP subject/performance Denotes an issue or pull request is related to replication performance. component/puller Puller component. labels Sep 27, 2020

overvenus reviewed Sep 27, 2020

View reviewed changes

liuzix added 3 commits September 28, 2020 14:05

sync

7086fe4

async

67ec0fd

sync

2145242

amyangfei mentioned this pull request Oct 10, 2020

Optimization performance of file sort #945

Closed

liuzix added status/ptal Could you please take a look? and removed status/WIP labels Oct 10, 2020

liuzix and others added 4 commits October 12, 2020 15:22

replace panics

c98002f

integration tests pending

66538e3

sync

b774a06

sync

02e98da

code format

a3c70c9

fix sorter unit test dir creation; fix github action image names

25d9ab7

Merge branch 'master' into zixiong-unified-sorter

1c94264

amyangfei reviewed Nov 25, 2020

View reviewed changes

amyangfei added this to the v4.0.9 milestone Nov 25, 2020

liuzix added 2 commits November 25, 2020 12:17

exclude testing code from code coverage

fdeb3c4

Merge branch 'zixiong-unified-sorter' of github.com:liuzix/ticdc into…

fa13fe9

… zixiong-unified-sorter

liuzix mentioned this pull request Nov 25, 2020

TiCDC: add information about Unified Sorter pingcap/docs-cn#4959

Merged

9 tasks

exclude testing utils from coverage

bc018aa

amyangfei reviewed Nov 25, 2020

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Update Makefile

01f8950

Co-authored-by: amyangfei <amyangfei@gmail.com>

amyangfei reviewed Nov 25, 2020

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 25, 2020

ti-srebot approved these changes Nov 25, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Nov 25, 2020

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2020

Merge branch 'master' into zixiong-unified-sorter

04f8d00

zier-one added the needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. label Nov 25, 2020

ti-srebot merged commit ad1354b into pingcap:master Nov 25, 2020

ti-srebot pushed a commit to ti-srebot/ticdc that referenced this pull request Nov 25, 2020

cherry pick pingcap#972 to release-4.0

cbfb230

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Nov 25, 2020

optimization: Unified Sorter (#972) #1122

Merged

amyangfei pushed a commit that referenced this pull request Nov 26, 2020

optimization: Unified Sorter (#972) (#1122)

3221296

* cherry pick #972 to release-4.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>

amyangfei mentioned this pull request Dec 1, 2020

Solution for replication with large lag of data #1150

Closed

6 tasks

ti-srebot mentioned this pull request Dec 21, 2020

TiCDC: add information about Unified Sorter (#4959) pingcap/docs-cn#5138

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optimization: Unified Sorter #972

optimization: Unified Sorter #972

liuzix commented Sep 27, 2020 •

edited

Loading

overvenus Sep 27, 2020

overvenus Sep 27, 2020

overvenus Sep 27, 2020

overvenus Sep 27, 2020

overvenus Sep 27, 2020

overvenus Sep 27, 2020

liuzix commented Oct 13, 2020

liuzix commented Oct 13, 2020

liuzix commented Oct 13, 2020

liuzix commented Nov 24, 2020

amyangfei commented Nov 25, 2020

liuzix commented Nov 25, 2020

amyangfei Nov 25, 2020

liuzix commented Nov 25, 2020

amyangfei left a comment

amyangfei left a comment

amyangfei commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

		heapSizeBytesEstimate += event.RawKV.ApproximateSize()
		if heapSizeBytesEstimate >= heapSizeLimit \|\| isResolvedEvent {

optimization: Unified Sorter #972

optimization: Unified Sorter #972

Conversation

liuzix commented Sep 27, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Benchmarks

Method

Results

Machine specifications

Sorter configuration

Release note

overvenus Sep 27, 2020

Choose a reason for hiding this comment

overvenus Sep 27, 2020

Choose a reason for hiding this comment

overvenus Sep 27, 2020

Choose a reason for hiding this comment

overvenus Sep 27, 2020

Choose a reason for hiding this comment

overvenus Sep 27, 2020

Choose a reason for hiding this comment

overvenus Sep 27, 2020

Choose a reason for hiding this comment

liuzix commented Oct 13, 2020

liuzix commented Oct 13, 2020

liuzix commented Oct 13, 2020

liuzix commented Nov 24, 2020

amyangfei commented Nov 25, 2020

liuzix commented Nov 25, 2020

amyangfei Nov 25, 2020

Choose a reason for hiding this comment

liuzix commented Nov 25, 2020

amyangfei left a comment

Choose a reason for hiding this comment

amyangfei left a comment

Choose a reason for hiding this comment

amyangfei commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

ti-srebot commented Nov 25, 2020

liuzix commented Sep 27, 2020 •

edited

Loading