Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimization: Unified Sorter #972

Merged
merged 91 commits into from
Nov 25, 2020
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
e7987be
initial commit
liuzix Sep 11, 2020
5b4d676
sync
liuzix Sep 14, 2020
94142c4
sync
liuzix Sep 14, 2020
caab34a
sync
liuzix Sep 17, 2020
55ceabb
sync
liuzix Sep 18, 2020
0c72c88
sync
liuzix Sep 21, 2020
a0e99cc
sync
liuzix Sep 23, 2020
29bfd04
sync
liuzix Sep 24, 2020
c1505c5
sync
liuzix Sep 24, 2020
bcd1713
sync
liuzix Sep 25, 2020
ee55414
sync
liuzix Sep 27, 2020
7086fe4
sync
liuzix Sep 28, 2020
67ec0fd
async
liuzix Sep 29, 2020
2145242
sync
liuzix Sep 30, 2020
c98002f
replace panics
liuzix Oct 12, 2020
66538e3
integration tests pending
liuzix Oct 10, 2020
b774a06
sync
liuzix Oct 13, 2020
02e98da
sync
liuzix Oct 13, 2020
a3c70c9
code format
liuzix Oct 13, 2020
25d9ab7
fix sorter unit test dir creation; fix github action image names
liuzix Oct 13, 2020
9a85856
unit test assertion
liuzix Oct 13, 2020
31d8f1f
try to fix data race
liuzix Oct 13, 2020
7351f0d
update default sorting engine
liuzix Oct 13, 2020
e456264
use errgroup
liuzix Oct 13, 2020
0e3a13c
update gitignore
liuzix Oct 13, 2020
fbccdc5
one backEndPool for all sorters
liuzix Oct 14, 2020
4cdf157
code format
liuzix Oct 14, 2020
7054e93
send event to mounter after sorting
liuzix Oct 14, 2020
7d7fd8d
create finished channel in PolymorphicEvent
liuzix Oct 14, 2020
3f86031
fix data race in fatal
liuzix Oct 14, 2020
6e60299
Merge remote-tracking branch 'origin/master' into zixiong-unified-sorter
liuzix Oct 14, 2020
12bb8f7
fix bugs
liuzix Oct 14, 2020
c41b21e
log error when failing to remove temp file
liuzix Oct 14, 2020
519753d
fix
liuzix Oct 14, 2020
7c7fc25
improve file name
liuzix Oct 14, 2020
01097b5
fix Finished channel in PolymorphicEvent
liuzix Oct 14, 2020
fd7661d
msgpack omit empty
liuzix Oct 14, 2020
7def952
fix
liuzix Oct 14, 2020
551fbd3
adjust concurrent heap number
liuzix Oct 14, 2020
52fc311
fix resolved event ts
liuzix Oct 14, 2020
f2ba2b7
fix unit test
liuzix Oct 14, 2020
62a63d5
fix unit test
liuzix Oct 14, 2020
e4fde8d
modify according to github comments
liuzix Oct 22, 2020
404691e
Merge branch 'master' of github.com:pingcap/ticdc into zixiong-unifie…
liuzix Oct 22, 2020
995a027
fix memory policy
liuzix Oct 30, 2020
31beafc
fix reopen mode
liuzix Nov 2, 2020
e7d7b6c
fix close file
liuzix Nov 2, 2020
ce4bb3b
add configuration options
liuzix Nov 2, 2020
fdd5b0d
add benchmark results
liuzix Nov 3, 2020
87336bd
print error message on unified sorter exits
liuzix Nov 4, 2020
16ab040
Merge branch 'master' into zixiong-unified-sorter
liuzix Nov 4, 2020
c528069
do not flush on resolved event
liuzix Nov 5, 2020
ca29c1d
limit flush rate
liuzix Nov 5, 2020
4597143
add more error logs
liuzix Nov 6, 2020
3765706
add cancellation latency monitor
liuzix Nov 8, 2020
f6c3218
fix printError
liuzix Nov 8, 2020
267b4ee
prevent fd leaks
liuzix Nov 10, 2020
ad31274
WIP
liuzix Nov 11, 2020
3a1cdd9
WIP
liuzix Nov 16, 2020
f1bdc1f
refactor
liuzix Nov 16, 2020
a368bb2
WIP
liuzix Nov 16, 2020
f8cbe22
refactor completed
liuzix Nov 17, 2020
66aae55
Merge branch 'master' of github.com:pingcap/ticdc into zixiong-unifie…
liuzix Nov 17, 2020
a34bf6d
truncate file on opening for writing
liuzix Nov 17, 2020
7ae4d31
fix bugs
liuzix Nov 17, 2020
6abf07e
fix bug
liuzix Nov 17, 2020
67ebafd
fix uncessary logs
liuzix Nov 17, 2020
0263f08
metrics
liuzix Nov 18, 2020
74a298c
fix processor race
liuzix Nov 18, 2020
231844c
fix write after close
liuzix Nov 18, 2020
621248a
add resolveTicker
liuzix Nov 19, 2020
85c9ff8
WIP
liuzix Nov 19, 2020
7d3f2d8
fix unit test
liuzix Nov 19, 2020
c197405
fix unit test
liuzix Nov 19, 2020
8bcdb40
fix unit test
liuzix Nov 19, 2020
d8ef14a
fix metrics
liuzix Nov 19, 2020
9500ab8
disable unified sorter by default
liuzix Nov 19, 2020
4f784ae
fix unit test
liuzix Nov 19, 2020
1c92484
fix default sorter
liuzix Nov 19, 2020
24fefa0
disable race detector in Unified Sorter case
liuzix Nov 20, 2020
35eb78f
fix MakeFile
liuzix Nov 20, 2020
70312ed
Merge remote-tracking branch 'origin/master' into zixiong-unified-sorter
liuzix Nov 24, 2020
4241880
fix unit test
liuzix Nov 24, 2020
bd781a6
add max-message-bytes to integration test
liuzix Nov 24, 2020
252d06e
fix cleanup_process
liuzix Nov 24, 2020
1c94264
Merge branch 'master' into zixiong-unified-sorter
liuzix Nov 25, 2020
fdeb3c4
exclude testing code from code coverage
liuzix Nov 25, 2020
fa13fe9
Merge branch 'zixiong-unified-sorter' of github.com:liuzix/ticdc into…
liuzix Nov 25, 2020
bc018aa
exclude testing utils from coverage
liuzix Nov 25, 2020
01f8950
Update Makefile
liuzix Nov 25, 2020
04f8d00
Merge branch 'master' into zixiong-unified-sorter
ti-srebot Nov 25, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions cdc/model/kv.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.

//go:generate msgp

package model

import (
Expand All @@ -32,6 +34,7 @@ const (

// RegionFeedEvent from the kv layer.
// Only one of the event will be setted.
//msgp:ignore RegionFeedEvent
type RegionFeedEvent struct {
Val *RawKVEntry
Resolved *ResolvedSpan
Expand All @@ -53,25 +56,26 @@ func (e *RegionFeedEvent) GetValue() interface{} {

// ResolvedSpan guarantees all the KV value event
// with commit ts less than ResolvedTs has been emitted.
//msgp:ignore ResolvedSpan
type ResolvedSpan struct {
Span regionspan.ComparableSpan
ResolvedTs uint64
}

// RawKVEntry notify the KV operator
type RawKVEntry struct {
OpType OpType
Key []byte
OpType OpType `msg:"op_type"`
Key []byte `msg:"key"`
// nil for delete type
Value []byte
Value []byte `msg:"value"`
// nil for insert type
OldValue []byte
StartTs uint64
OldValue []byte `msg:"old_value"`
StartTs uint64 `msg:"start_ts"`
// Commit or resolved TS
CRTs uint64
CRTs uint64 `msg:"crts"`

// Additonal debug info
RegionID uint64
RegionID uint64 `msg:"region_id"`
}

func (v *RawKVEntry) String() string {
Expand Down
320 changes: 320 additions & 0 deletions cdc/model/kv_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading