Init support for filebeat multiline

[urso]

start with multiline for filebeat (#461)

[tsg]

I'd call the option multiline to be more clear what is about.

[ruflin]

I would suggest to have max_bytes outside multiline as general configuration, as it not only applies to multiline but could also apply to a very large single line.

[urso]

Good point. Will do. same for max_bytes.

[ruflin]

What you mean by same for max_bytes?

[urso]

I meant to say max_lines... but max_lines makes no sense. Just will do for max_bytes. Sorry for the noise.

[ruflin]

@urso Thanks a lot for getting this started. I had a closer look, but didn't test it yet locally. Some notes:

• As it is quite a big change, it is not easy to follow all the changes. One thing I'm very caution about to change is the error handling functionality as it has (too) much magic inside. Best on a brief comparison I assume it stayed mostly the same: https://github.com/elastic/beats/pull/570/files#diff-0df022979a0f5b03f3e4f709bd9c9c4dR98
• We should definitively add system tests. I can help you here
• Interestingly I would probably have followed a different approach for the implementation. Instead of moving the functionality and error handling to the line readers, I would have kept it on a higher level. I would probably add functionality to the Harvest function. The main reason for this as I would argue the line readers should not have to deal with timeouts, backoffs or the error handling. Looking at the implementation it could be that it is more a "naming" discussion, as the basic line by line reader still exist. This is probably easier to discuss via chat with a small diagram :-)

[urso]

Error handling did not change. But I found error handling or retry functionallty to be too much inter winded. I did try to build "small" composable units being responsible for just one functionality when reading lines. I did find there is no 'higher' level without abstraction leak or doing all multiline-handling much more inter winded with error handling.
So I pushed retry logic + backoff to an earlier stage so not to confuse the multiline-reader state machine. I didn't want to handle multiline in main-loop, which is already hard to follow.

Processing stages (pipe-line is setup as follows):

file input (io.Reader) ->
logFileReader (io.Reader with retry and backoff) ->
encoding.LineReader (read lines and apply codec) ->
[optional] timeoutReader (force multiline Reader to publish current state) ->
[optional] multilineReader (state-machine combining multiple lines) ->
noEOLLineReader (remove trailing newline) ->
harvester loop (publish line event and quit loop on error)

The logFileReader implements error handling on lower level, so state machines in LineReader and multilineReader have sane error handling + timeout reader can be safely shutdown if error is not nil. Having the logFileReader removes the requirement of readers being still valid after EOF.

[urso]

In this PR I've implemented a layered approach:

1. raw input layer
2. processing layer (codec, line reader, multiline, remove trailing newline)
3. event publisher loop

Layers 1 and 2 use the io.Reader interface and layers 2 and 3 use the lineReader interface. By swapping the input layer, all processing layers can be reused for different inputs (e.g. TCP/unix socket or stdin). I think layers for dealing with complexity + composing processing elements by interfaces is a quite powerful and easy to use mechanism that fits well with i/o interfaces in go stdlib. Some more refactoring + cleanup in the harvester is required in general.

It is the input layer which must behave different for different source types. For log files we want backoff + retry on EOF, but not for sockets or stdin. It's up to the input layer to push bytes until some well-defined EOF is reached. For log files the input layer is implemented by logFileReader. Sockets and file handle for stdin can be used as is (we don't want to ignore EOF). The error handling as it has been implemented in log.go was a good fit for log files, but not so much for stdin for example (which it was used for).

There is no real error handling in the line readers, but I have had to move some error handling to the input layer, so statefull processors in layer 2 are not affected by error handling/recovery in layer 3.

I think another advantage of having the logFileReader handling the reading from files is, we're more flexible in adding file reading features. For example:

• on windows close the file after EOF and retry (by re-opening the file) after backoff seconds. No state in upper layers will be affected
• have central authority watching number of open files and schedule temporary closing/opening of files in logFileReader in order to limit the total number of file descriptors used by filebeat.

[ruflin]

Should be maxBytes part of logFileReaderConfig? Same for bufferSize and codec?

[urso]

logFileReadConfig is the input layer config (file reading only). The other options codec, bufferSize and maxBytes are used to configure the line processing layer.

[ruflin]

Typo: s/hast/has/

[andrewkroh]

I think Pattern can be lower-cased here. Do you think we should say that it is the same syntax as Perl or link to https://github.com/google/re2/wiki/Syntax.

[urso]

we're using CompilePOSIX to compile the regular expressions. These are 'mostly' compatible to egrep + some quirks (see documentation).

TBH I find it quit difficult to understand the impact on syntax when given this line in documentation:

restricts the regular expression to POSIX ERE (egrep) syntax

[urso]

even worse, you have to keep YAML string escaping in mind when doing the regular expression

[ruflin]

That also relates the the Regexp expression for exclude and include, right? If yes, I suggest to create a separate page with docs to regexp we can link to.

[urso]

Yes, we use same compile function.

[ruflin]

Should we open an issue for that to not forget about it and discuss it with @dedemorton ?

[ruflin]

LGTM

[andrewkroh]

s/append/appended/

[andrewkroh]

s/hast/has