-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add pending control to batcher #4042
Conversation
658eba1
to
6035acd
Compare
// NewPointBatcher returns a new PointBatcher. | ||
func NewPointBatcher(sz int, d time.Duration) *PointBatcher { | ||
// NewPointBatcher returns a new PointBatcher. sz is the batching size, | ||
// bf is the maximum number of batches that may be pending. d is the time |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/bf/bp/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ha, I used to call it bf
in the code, but forgot to update the comment -- thanks.
Small comment issue but 👍 |
With this change, the generic batcher used by many inputs can now be buffered. Testing shows that this performance of the Graphite input by 10-100%, with the biggest improvements at lower numbers of connections.
6035acd
to
24aca56
Compare
Hello, Many thanks for this change and your continual improvements - every new version generally keeps getting better 👍 Have been running this version in our testing environment and it looks much, much better write performance to graphite plugin wise. We have not seen a client write failure since the upgrade and CPU usage of InfluxDB has dropped a bit on average as well. There is only one connection in to the influxdb graphite plugin. Client logs: [2015-09-10 10:46:46] failed to write() to 127.0.0.1:2013: Connection reset by peer
[2015-09-10 10:46:58] server 127.0.0.1:2013: OK
[2015-09-10 10:50:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 10:50:36] server 127.0.0.1:2013: OK
[2015-09-10 10:50:38] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 10:50:38] server 127.0.0.1:2013: OK
[2015-09-10 11:12:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:12:36] server 127.0.0.1:2013: OK
[2015-09-10 11:13:38] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:13:38] server 127.0.0.1:2013: OK
[2015-09-10 11:17:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:17:36] server 127.0.0.1:2013: OK
[2015-09-10 11:17:38] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:17:38] server 127.0.0.1:2013: OK
[2015-09-10 11:33:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:33:36] server 127.0.0.1:2013: OK
[2015-09-10 11:34:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:34:36] server 127.0.0.1:2013: OK
[2015-09-10 11:36:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:36:36] server 127.0.0.1:2013: OK
[2015-09-10 11:39:35] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:39:36] server 127.0.0.1:2013: OK
[2015-09-10 11:40:36] failed to write() to 127.0.0.1:2013: uncomplete write
[2015-09-10 11:40:36] server 127.0.0.1:2013: OK
[2015-09-10 12:16:20] failed to write() to 127.0.0.1:2013: Connection reset by peer
[2015-09-10 12:17:28] server 127.0.0.1:2013: OK 12:16:20 was when the DB was upgraded - no write errors since then. |
Great, thanks for the feedkback @pkittenis |
With this change, the generic batcher used by many inputs can now be buffered. Testing shows that this improves performance of the Graphite input by 10-100%, with the biggest improvements at lower numbers of connections. Internal testing showed that this configuration -- a batcher per service, as opposed to per connection -- gave the best throughput and stable behaviour.
This patch also sets defaults for UDP input batching, which were not being set before.