Throttle import #3795
Merged
Throttle import #3795
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -146,7 +162,7 @@ func (i *Importer) processDML(scanner *bufio.Scanner) { | |||
| if strings.HasPrefix(line, "#") { | |||
| continue | |||
| } | |||
| i.batchAccumulator(line) | |||
| i.batchAccumulator(line, start) | |||
| } | |||
There was a problem hiding this comment.
Very minor, but I wouldn't bother with the variable. i.batchAccumulator(line, time.Now()). Purely cosmetic.
There was a problem hiding this comment.
This isn't cosmetic. It has to be at the beginning before the loop or it won't be able to calculate the total time lapsed.
There was a problem hiding this comment.
Ah, OK, I misinterpreted the code. Thanks @corylanou
|
OK, I see how it works. There are lots of ways to do this kind of thing, this works. +1 |
|
cherry picked into 0.9.3 |
|
Yes pls! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the ability to throttle the import.
Currently, unthrottled on my machine it tops out at about 160k points per second importing. I have not been able to saturate the system so it may be that some people won't need this option.
To throttle, you use the following command:
Which is stating that you don't want MORE than 50,000 points per second to write to the database. Due to the processing that is taking place however, you will likely never get exactly 50,000 pps, more like 35,000 pps, etc.
I added a status message as well that will give you stats on the progress of the import that shows up like this: