Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use two goroutines to speed up "load data" #11517

Closed
ngaut opened this issue Jul 30, 2019 · 1 comment
Closed

Use two goroutines to speed up "load data" #11517

ngaut opened this issue Jul 30, 2019 · 1 comment
Assignees
@cfzjywxk
Labels
type/performance

Comments

@ngaut
Copy link
Member

ngaut commented Jul 30, 2019

According to

tidb/server/conn.go

Line 1057 in 77b6858

err := loadDataInfo.CheckAndInsertOneBatch()

There are two steps for "load data" statement, we can use two goroutine to speed up the process, one goroutine for preparing data, and another goroutine for committing.
@ngaut ngaut added type/question The issue belongs to a question. type/performance and removed type/question The issue belongs to a question. labels Jul 30, 2019
@ngaut ngaut changed the title Use two goroutine to speed up "load data" Use two goroutines to speed up "load data" Jul 30, 2019
@cfzjywxk cfzjywxk self-assigned this Jul 30, 2019
@cfzjywxk
Copy link
Contributor

will do some experiments on this. Things need to consider

  • consume memory resources, especailly when commit is slow
  • load data statement may be changed to atomic like innodb in the future, this optimization may not work

@cfzjywxk cfzjywxk mentioned this issue Jul 30, 2019
Merged
@ngaut ngaut closed this as completed Nov 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cfzjywxk cfzjywxk

Labels
type/performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ngaut @cfzjywxk