Filebeat refactoring / restructuring

[ruflin]

Filebeat must be refactored / restructured to make event publishing less complex and adding new prospector types easier.

• Send events from harvester directly to spooler instead of through prospector Refactor harvester to send events directly to the spooler  #4070
  • Unify data handling across filebeat Unify event, data and state handling in filebeat #4171
  • Move json field handling to log harvester Move handling of json fields to harvester #4159
• Extract stdin prospector from prospector package Extract stdin prospector #4158
• Extract log prospector from prospector package, cleanup naming Move prospector log to its own package #4273
• Generalise state handling to make it available to other prospector types Make state comparison more generic #4190
  • Add type to registry to handle states of other prospectors Add type to state info #4185
  • Abstract state so each prospector can have different or not state object. The exact format should not matter for the registrar.
• Rename input_type to just type in config Rename input_type to type in config and input_type to prospector.type in event #4294
• Decouple stdin and log harvester. Currently stdin uses most of the logic from the log harvester which makes these two packages overlapping.
• Replace old publisher with new and move processors to pipeline
• Think about future data model of prospector types. See Filebeat Redis prospector type #4180 as an example. Will we have harvester types?
• Cleanup channel / outlet handling across filebeat. Currently it all depends too much on Outleter.
• Store fields information in _meta directory with each prospector type and collect it from there.

[ruflin]

@ph Pinging you on this one as I'm not sure you are aware of it. Perhaps we can also close this issue by now?

[ph]

Oh Lets close it, I didn't about this issue :)