Skip to content

Commit

Permalink
Make it clear in the docs that publish_async is still experimental (e…
Browse files Browse the repository at this point in the history
…lastic#3096)

Remove example for publish_async from the docs
(cherry picked from commit cbab48d)
  • Loading branch information
ruflin committed Dec 1, 2016
1 parent a316c6d commit 8c6da76
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 27 deletions.
31 changes: 4 additions & 27 deletions filebeat/docs/load-balancing.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
== Load Balancing

Filebeat provides configuration options that you can use to fine
tune load balancing when sending events to multiple hosts.
tune load balancing when sending events to multiple hosts.

To enable load balancing, you specify `loadbalance: true` when you configure
the output. For example:
Expand Down Expand Up @@ -32,7 +32,7 @@ The load balancer also supports multiple workers per host. The default is
connections will be used. The total number of workers participating
in load balancing is `number of hosts * workers`.
+
Example:
Example:
+
[source,yaml]
-------------------------------------------------------------------------------
Expand All @@ -46,15 +46,15 @@ output.logstash:
worker: 2
-------------------------------------------------------------------------------
+
In this example, there are 4 workers participating in load balancing.
In this example, there are 4 workers participating in load balancing.

* **Send events to `N` hosts in lock-step:**
+
You can configure Filebeat to send events to `N` hosts in lock-step by setting
`spool_size = N * bulk_max_size`. In lock-step mode, the batch collected by the
spooler is split up into smaller batches of size `bulk_max_size`. These smaller
batches are load balanced between available connections. Filebeat waits for all
sub-batches to be published before it retrieves another batch from the spooler.
sub-batches to be published before it retrieves another batch from the spooler.
+
This mode requires more memory and CPU usage than the previous mode.
+
Expand All @@ -72,26 +72,3 @@ output.logstash:
loadbalance: true
bulk_max_size: 2048
-------------------------------------------------------------------------------

* **Send events in parallel and asynchronously:**
+
You can configure Filebeat to send events in parallel and asynchronously by
setting `publish_async: true`. With this setting, Filebeat pushes a batch of
lines and then prepares a new batch of lines while waiting for the output to
ACK. This mode can improve load-balancing throughput, but requires the most
memory and CPU usage.
+
Example:
+
[source,yaml]
-------------------------------------------------------------------------------
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*.log
filebeat.publish_async: true
output.logstash:
hosts: ["localhost:5044", "localhost:5045"]
loadbalance: true
-------------------------------------------------------------------------------

Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,8 @@ See <<load-balancing>> for more information about how this setting affects load

===== publish_async

experimental[]

If enabled, the publisher pipeline in Filebeat operates in async mode preparing
a new batch of lines while waiting for ACK. This option can improve load-balancing
throughput at the cost of increased memory usage. The default value is false.
Expand Down
2 changes: 2 additions & 0 deletions filebeat/publisher/publisher.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (

"github.com/elastic/beats/filebeat/input"
"github.com/elastic/beats/libbeat/common"
"github.com/elastic/beats/libbeat/logp"
"github.com/elastic/beats/libbeat/publisher"
)

Expand Down Expand Up @@ -33,6 +34,7 @@ func New(
pub publisher.Publisher,
) LogPublisher {
if async {
logp.Warn("Using publish_async is experimental!")
return newAsyncLogPublisher(in, out, pub)
}
return newSyncLogPublisher(in, out, pub)
Expand Down

0 comments on commit 8c6da76

Please sign in to comment.