[Filebeat][CometD] Filebeat CometD input closes Filebeat worker while getting connection or EOF error from bayeux library

[kush-elastic]

Description:

Filebeat CometD input is closing input worker in case of client-server connection issue with EOF error.
Current forked bayeux library is sending error message while any connection issues from salesforce side. which in return CometD is assuming it to be un recoverable error and closing the input worker in the Filebeat.

Logs:

{"log.level":"error","@timestamp":"2023-01-20T16:17:09.542+0530","log.logger":"cometd","log.origin":{"file.name":"cometd/input.go","file.line":46},"message":"got error while running input","service.name":"filebeat","pubsub_channel":"channel_name","error":{"message":"error collecting events: cannot connect to bayeux: bad unrecoverable call: Post "http://localhost:8080/cometd/38.0\": EOF, trying again"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-20T16:17:09.543+0530","log.logger":"cometd","log.origin":{"file.name":"cometd/input.go","file.line":47},"message":"Input worker has stopped.","service.name":"filebeat","pubsub_channel":"channel_name","ecs.version":"1.6.0"}

Solution:

Need to manage errors or in this case can be called warnings should be explicitly handled to not close the input and continue trying to collecting data from CometD.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic]

This issue doesn't have a Team:<team> label.

[kush-elastic]

CometD input does not have restart on failures. which is case worker is not getting restarted even if user has not explicitly closed Filebeat.

Solution:

• Add worker context closer condition to restart worker each time if user has not sent signal for closing the filebeat(which here can be checked using worker context).
• Add channel handler for each iteration.
• Add rate limit with wait time to each restart of input worker.