Live reload config files without restarting filebeat process

[superwhykz]

Based on discussion opening enhancement feature request.
https://discuss.elastic.co/t/reload-if-modification-time-changes-in-config-files/55556

[slava-vishnyakov]

It would probably be better to reload, like nginx with something like kill -s HUP

Although logstash, for ex., uses automatic reload https://www.elastic.co/guide/en/logstash/current/reloading-config.html

[OferE]

I'm working with microservices architecture and restarting the filebeat process is a big pain to me,
I have many services coming and going and i want them to register themselves with filebeats without restarting the filebeat process everytime a service joins.
Restart a process is a very aggressive operation and has consequences regarding the exactly once guarantees.

[ruflin]

Does every service need its own config / prospector? Or do you know all of them in advance and could provide all the prospectors on first start?

[OferE]

knowing all services in advance is a bad practice since i have multiple machine types, each type runs a different set of services. each machine may run different services even with the same machine type.
currently this is how i workaround everything: by providing one huge config.
The correct approach IMHO will be that every service will register itself with its own config - the current filebeat process will reload the configuration on every change/inroduction of new config and adjust in a clean way.
By the way - the outputs themselves should have the ability to change dynamically and accrding to each prospector, since some services may want to write to kafka (even into different topics) and others to elasctic/logstash.
This should be per prospector and not as a global definition.
The world is moving into microservices architectures - filebeat is a great project - but it is not a perfect match for micro services.

[ruflin]

@OferE I was also more thinking of current work arounds, but it seems you found one. It is definitively something we are looking into also related to #464

[OferE]

@ruflin - great to see that i am useful - also great to see that u hear community feedback! great product - i tend to use it in production for streaming messages to Kafka un huge scale.

[aqiao]

hi guys, may i know the latest progress of this feature?

[ruflin]

@aqiao Interesting timing. There is not directly a filebeat update but I just pushed this PR here: #3281 This could be reused for filebeat prospectors in the future. It's a little bit trickier for filebeat as it must be ensured that a file is only harvested once and brings up the question when exactly a harvester should shut down ...

[ruflin]

Closing as #3362 was merged to master. This allows to dynamically reload prospectors.

[OferE]

@ruflin thanks for this feature!