Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial version of craft binary mq message format (#1621) #2130

Closed
wants to merge 39 commits into from
Closed

Initial version of craft binary mq message format (#1621) #2130

wants to merge 39 commits into from

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #1621

Signed-off-by: Xiaoguang Sun sunxiaoguang@zhihu.com

What problem does this PR solve?

#1613

What is changed and how it works?

Implement a binary MQ format that's under discussion in the issue #1613

Check List

Tests

  • Unit test

Code changes

  • Has persistent data change

Side effects

  • Increased code complexity

Related changes

  • Need to update the documentation

Release note

  • sink: Add a new binary MQ format that is specially designed for TiDB and more compact than JSON base Open Protocol. In addition to footprint and performance benefits. There is Flink format plugin support with this new format which makes it possible to use TiDB as a unified batch / streaming source connector in Flink.

@sunxiaoguang @ti-chi-bot
Initial version of craft binary mq message format
47f4bee 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Add license
c69c353 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix some lint
fab036b 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Add more test to supress not used function lint
af5f8c8 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Remove not useful decodeRowChangedEvent function
b14a8e7 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Use the correct size table to decode column groups
c4528d3 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix more lints
0472d8d 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
955cb7c 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
201f90b 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Change to little endian like what protobuf does.
a318a49 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Remove comments and fix column group values encoding
cba72ba 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Set schema and table to nil when they are empty
8b157fe 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Add more test cases and fix bugs about the fail tests
c8ee55e 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Add protobuf vs craft benchmark related code
e4095af 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix column group & year type seralization bug
6992aaf 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Force encoding year type to varint
8c37b36 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix body size table
34f0ded 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Disable lint for not yet used utility functions
a9a27d4 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
7d1b4f2 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
b640062 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
e631aec 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
a6a3cd0 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lint
f1033d8
@sunxiaoguang @ti-chi-bot
Remove unnecessary assignment
3d6e15e 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix errdoc
c4bda42 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Change decoder to use body size table instead of offset table
5611ddf 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix json.Number.Int64() overflow
bf063bc 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Convert from float64 to int64 or uint64
7fab547
@sunxiaoguang @ti-chi-bot
Use c.Logf instead of fmt.Printf
e7359bf 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Seed random number generator in init function
205937d 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Add comments to varint encoding logic
9bfefe6 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix couple of typos and add some comments
ad97b40 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Remove deprecated RowID from header
3ad6753 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Optimize codec size
f9d5004 
1. Remove deprecated RowID field from header
2. Use term dictionary to compress terms

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Fix lints
ed66033 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Change size table to encode with varint chunk
12c69ff 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Always encode terms in delta varint chunk
5abe993 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Change header to encode type with delta varint chunk
d519db7 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang @ti-chi-bot
Use uvarint chunk to encode type of headers
fa25abf 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@ti-chi-bot
Copy link
Member Author

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot mentioned this pull request Jun 22, 2021
Merged
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-5.1 This PR is cherry-picked to release-5.1 from a source PR. labels Jun 22, 2021
@sunxiaoguang
Copy link
Contributor

/run-all-tests

@amyangfei amyangfei added this to the v5.1.1 milestone Jun 23, 2021
@Mini256 Mini256 mentioned this pull request Jul 9, 2021
Closed
1 task
@lonng lonng added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Jul 14, 2021
@amyangfei amyangfei modified the milestones: v5.1.1, v5.1.2 Jul 30, 2021
@amyangfei
Copy link
Contributor

amyangfei commented Sep 2, 2021 •
edited
Loading

This feature is released in v5.2.0 (ref: TiDB 5.2 Release Notes), and we don't need to pick it to release version.

@amyangfei amyangfei closed this Sep 2, 2021
@sunxiaoguang sunxiaoguang mentioned this pull request Sep 13, 2021
Open
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@IANTHEREAL IANTHEREAL Awaiting requested review from IANTHEREAL

@zier-one zier-one Awaiting requested review from zier-one

@liuzix liuzix Awaiting requested review from liuzix

@overvenus overvenus Awaiting requested review from overvenus

Assignees
No one assigned
Labels
cherry-pick-approved Cherry pick PR approved by release team. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-5.1 This PR is cherry-picked to release-5.1 from a source PR.
Projects
None yet
Milestone
v5.1.2
Development

Successfully merging this pull request may close these issues.
4 participants
@ti-chi-bot @sunxiaoguang @amyangfei @lonng