Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc goroutines become crazy when initializing a capture with lots of regions and tikv.cdc.min-ts-interval = 200ms #5610

Closed
hicqu opened this issue May 27, 2022 · 3 comments · Fixed by #5658
Closed

cdc goroutines become crazy when initializing a capture with lots of regions and tikv.cdc.min-ts-interval = 200ms #5610

hicqu opened this issue May 27, 2022 · 3 comments · Fixed by #5658
Assignees
@hicqu
Labels
affects-5.4 affects-6.1 area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@hicqu
Copy link
Contributor

hicqu commented May 27, 2022 •
edited
Loading

What did you do?

Create a capture on 16 MySQL tables, 131K regions totally with

  • tikv.cdc.min-ts-interval = 200ms
  • tikv.cdc.min-ts-interval = 1s

What did you expect to see?

Both 2 operations can be finished quickly. With tikv.cdc.min-ts-interval = 1s it's expected:
图片

What did you see instead?

However with tikv.cdc.min-ts-interval = 200ms the capture creating can never finished, and cpu usage is very high:
图片
图片

And here is a 30s CPU profile:
图片

We can see the CPU time is about 480s, which means all 16C are very busy on runtime.selectgo.

Versions of the cluster

All components are nightly.
Go version is 1.18.1.

Investigations
@hicqu hicqu added type/bug The issue is confirmed as a bug. area/ticdc Issues or PRs related to TiCDC. labels May 27, 2022
@hicqu hicqu self-assigned this May 27, 2022
@hicqu hicqu mentioned this issue May 30, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Jun 8, 2022
@hicqu
cdc: reduce context switch for stateful events (#5658)
680740c 
close #5610
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Jun 8, 2022
@hicqu @3AceShowHand
cdc: reduce context switch for stateful events (pingcap#5658)
2a40e0d 
close pingcap#5610
sdojjy pushed a commit to sdojjy/tiflow that referenced this issue Jul 19, 2022
@hicqu @sdojjy
cdc: reduce context switch for stateful events (pingcap#5658)
cf13207 
close pingcap#5610
# Conflicts:
#	cdc/kv/client.go
#	cdc/kv/region_worker.go
sdojjy added a commit to sdojjy/tiflow that referenced this issue Jul 19, 2022
@sdojjy
cdc: reduce context switch for stateful events (pingcap#5658)
519b01e 
close pingcap#5610
# Conflicts:
#	cdc/kv/client.go
#	cdc/kv/region_worker.go
sdojjy pushed a commit to sdojjy/tiflow that referenced this issue Jul 20, 2022
@hicqu @sdojjy
cdc: reduce context switch for stateful events (pingcap#5658)
81fd4be 
close pingcap#5610
# Conflicts:
#	cdc/kv/client.go
#	cdc/kv/region_worker.go
@nongfushanquan
Copy link
Contributor

/label affect-5.4

@ti-chi-bot
Copy link
Member

@nongfushanquan: The label(s) affect-5.4 cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label affect-5.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@nongfushanquan
Copy link
Contributor

/label affects-5.4
/label affects-6.1

This was referenced Jul 20, 2022
Merged
Closed
This was referenced Jul 20, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Jul 21, 2022
@sdojjy
puller(ticdc): hotfix for 5.4.1- reduce context switch for stateful e…
687234d 
…vents (#6385)

close #5610
@hicqu hicqu mentioned this issue Jul 21, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Jul 21, 2022
@hicqu
kv(cdc): set enableOldValue correctly (#6410)
cbee125 
ref #5610
ti-chi-bot added a commit that referenced this issue Jul 25, 2022
@ti-chi-bot
cdc: reduce context switch for stateful events (#5658) (#6375)
455ba52 
close #5610
This was referenced Aug 29, 2022
Merged
Merged
ti-chi-bot pushed a commit that referenced this issue Sep 14, 2022
@sdojjy
puller(ticdc): cherry-pick 5658 to release-5.4 (#6377)
2172d71 
ref #5610
@nongfushanquan nongfushanquan mentioned this issue Oct 9, 2022
Merged
12 tasks
@nongfushanquan nongfushanquan mentioned this issue Oct 9, 2022
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hicqu hicqu

Labels
affects-5.4 affects-6.1 area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cdc: reduce context switch for stateful events hicqu/ticdc
3 participants
@hicqu @ti-chi-bot @nongfushanquan