Lumberjack V2 output

[robin13]

• Version: 3.1.7

This plugin currently communicates only using Lumberjack V1 output, but this is not compatible with the Lumberjack V2 protocol. This can be problematic if a custom application (not logstash-input-beats which speaks both V1 and V2) has been designed to work with the Lumberjack V2 protocol only.
It would be great to have an option in the logstash-output-lumberjack plugin to choose the protocol version being used for communication.

[webmat]

Note to future implementer, Lumberjack protocol V1 is explained here.

According to @ph, V2 differences are:

Just to make sure we don't lose reference to this, the protocol defined here contains accurate information about v1.
In short, V2 is replaced with V1 and the payload is in the form of key/value pair instead of using a JSON format.
The ACK in V1 is a bit different:
Do not support ACK0 keep alive.
Do no support partial ACK
Only ACK on full batch

[webmat]

@ph @jordansissel Is there a more formally defined spec for Lumberjack V2? Or is the Go implementation in Beats the only reference?

[ntim]

Lumberjack v2 also works better in combination with custom load balancers.

Furthermore, when configuring logstash-to-logstash communication as shown in the official docs, every minute or so, the sending logstash complains about a malformed frame.

[ntim]

@ph @jsvd @jordansissel any updates?

[ntim]

@webmat documentation of the protocol will not be done: https://github.com/elastic/libbeat/issues/279