Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink: Pass extra meta about the event to MQ producer #1442

Merged
merged 14 commits into from
Mar 1, 2021
Merged

sink: Pass extra meta about the event to MQ producer #1442

merged 14 commits into from
Mar 1, 2021

Conversation

sunxiaoguang
Copy link
Contributor

Pass extra meta about the event to mq producer

What problem does this PR solve?

Pass extra structured meta data (Ts, Schema, Table) for event in Context passed to MQ producer. We can use these extra meta data in message for application to seek quickly from messages in the past.

What is changed and how it works?

Put meta data about events in MQMessage and change MQ Producer interface to accept message instead of key and value slices.

Check List

Tests

  • Unit test

Code changes

  • Has exported variable/fields change
    Add Schema, Table, Type and Protocol fields to MQMessage

  • Has interface methods change
    Change Producer to accept MQMessage instead of key and value slices

Side effects

Related changes

Release note

  • No release note

sunxiaoguang and others added 4 commits February 23, 2021 14:44
@sunxiaoguang
Pass extra meta about the event to mq producer
a7be3a2 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang
Add unit test for interface changes
7a2de53
@sunxiaoguang
Merge branch 'master' into event_meta
c89e4f6
@sunxiaoguang
Fix kafka producer test
518977f 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang
Fix interface test
c6ad5a1 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@sunxiaoguang
Add testleak check
c21e3ae 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@liuzix liuzix changed the title [sink] Pass extra meta about the event to mq producer sink: Pass extra meta about the event to MQ producer Feb 23, 2021
@liuzix
Copy link
Contributor

liuzix commented Feb 24, 2021

/run-all-tests

2 similar comments
@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@codecov-io
Copy link

codecov-io commented Feb 26, 2021 •
edited
Loading

Codecov Report

Merging #1442 (574b061) into master (d36ae83) will decrease coverage by 0.0097%.
The diff coverage is 64.2512%.

@@               Coverage Diff                @@
##             master      #1442        +/-   ##
================================================
- Coverage   47.8373%   47.8275%   -0.0098%     
================================================
  Files           145        146         +1     
  Lines         14727      14845       +118     
================================================
+ Hits           7045       7100        +55     
- Misses         6946       7009        +63     
  Partials        736        736

@amyangfei amyangfei added status/ptal Could you please take a look? needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. component/sink Sink component. labels Feb 26, 2021
sunxiaoguang and others added 2 commits February 28, 2021 13:30
@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@sunxiaoguang
Copy link
Contributor Author

sunxiaoguang commented Mar 1, 2021 •
edited
Loading

#1431

overvenus
overvenus reviewed Mar 1, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cdc/sink/codec/interface.go Outdated
Comment on lines 74 to 75
ms := int64(m.Ts >> physicalShiftBits)
return time.Unix(ms/1e3, (ms%1e3)*1e6)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use github.com/pingcap/tidb/store/tikv/oracle.GetTimeFromTS.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, It sounds reasonable to reuse code from TiDB.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 1, 2021
amyangfei
amyangfei reviewed Mar 1, 2021
View reviewed changes
cdc/sink/codec/interface.go Outdated
Comment on lines 74 to 75
ms := int64(m.Ts >> physicalShiftBits)
return time.Unix(ms/1e3, (ms%1e3)*1e6)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be simplified with oracle.GetTimeFromTS(m.Ts)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, It sounds reasonable to reuse code from TiDB.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

cdc/sink/codec/interface_test.go Outdated
@@ -0,0 +1,121 @@
// Copyright 2020 PingCAP, Inc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Copyright 2020 PingCAP, Inc.
// Copyright 2021 PingCAP, Inc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@sunxiaoguang
Fix copyright and reuse TSO util function from TIDB
e856643 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
amyangfei
amyangfei reviewed Mar 1, 2021
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Mar 1, 2021
@ti-srebot ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Mar 1, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-srebot ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 1, 2021
@ti-srebot
Copy link
Contributor

Your auto merge job has been accepted, waiting for:

  • 1461

@sunxiaoguang
Remove not useful const
574b061 
Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
@ti-srebot
Copy link
Contributor

/run-all-tests

@zier-one
Copy link
Contributor

zier-one commented Mar 1, 2021

/merge

@ti-srebot
Copy link
Contributor

Your auto merge job has been accepted, waiting for:

  • 1437
  • 1461

@ti-srebot
Copy link
Contributor

@sunxiaoguang merge failed.

@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor

/merge

@ti-srebot
Copy link
Contributor

Your auto merge job has been accepted, waiting for:

  • 1461
  • 1437
  • 1461

@sunxiaoguang
Copy link
Contributor Author

/merge

@ti-srebot
Copy link
Contributor

Your auto merge job has been accepted, waiting for:

  • 1461
  • 1437
  • 1461

@sunxiaoguang
Copy link
Contributor Author

/run-all-tests

1 similar comment
@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot
Copy link
Contributor

@sunxiaoguang merge failed.

@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot ti-srebot merged commit 08d535c into pingcap:master Mar 1, 2021
ti-srebot pushed a commit to ti-srebot/ticdc that referenced this pull request Mar 1, 2021
@sunxiaoguang @ti-srebot
cherry pick pingcap#1442 to release-4.0
3da202a 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Mar 1, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #1466

@sunxiaoguang sunxiaoguang deleted the event_meta branch March 2, 2021 04:37
ti-srebot added a commit that referenced this pull request Mar 2, 2021
@ti-srebot
sink: Pass extra meta about the event to MQ producer (#1442) (#1466)
93b5247 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei left review comments

@overvenus overvenus overvenus left review comments

@ti-srebot ti-srebot ti-srebot approved these changes

Assignees
No one assigned
Labels
component/sink Sink component. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look?
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@sunxiaoguang @liuzix @codecov-io @amyangfei @ti-srebot @zier-one @overvenus