Concurrency issues with Log4J2 Rolling File Appender

[Patrick-Harms]

I have a server application that consists of many separate processes (sometimes 20+) that use Log4J2 for logging and based on the number need to log to a central Log4J2 rolling file appender. A rolling file appender, as per usual is needed, to keep on top of disc space consumed by logging.

The "out of the box" Log4J2 rolling file appender has concurrency issues, when multiple processes (and I have caused issues with just two instances).

The issue, at initial investigation, is that all the processes want to be Master, and roll the file. That causes an error, and on the next log message, the process then gets a hard error that the stream is closed.

Does anyone know of a configuration sharing the same destination log file?

Or do I need to write my own custom appender to handle concurrency issues?

[swebb2066]

Have you considered sending logging events to the TCP input plugin of a fluent-bit log server?

I have observed in Log4cxx that coordinating the rollover of the file adds a large overhead (4 and 10 times on Linux and Windows respectively) to each logging request

[ppkarwasz]

@Patrick-Harms,

The issue, at initial investigation, is that all the processes want to be Master, and roll the file. That causes an error, and on the next log message, the process then gets a hard error that the stream is closed.

The rolling file appender does not support logging to the same file from separate JVMs, because there is no reliable mechanism in Java to decide which process will be the "primary" process, i.e. will actually perform the rotation. The same log file can only be reused by multiple applications in the same application server (see note on RollingFileManager sharing).

Even if you solve the problem of files being rotated multiple times, other features of the rolling file appender will still not work with multiple JVMs: e.g. the SizeBasedTriggeringPolicy will not take into account the logs from other JVMs to determine the current size of the log file.

As mentioned by @swebb2066, if you want to use the same files in multiple applications, you should forward all the logs to an external application, like the mentioned Fluent Bit, Fluentd or the common RSyslog. Note that Log4j Core has specialized RFC 5424 and RFC 3164 layouts to format log events in a way recognized by Syslog. You can combine them with a file or socket appender to send messages to RSyslog.

[Patrick-Harms]

Thanks @swebb2066 and @ppkarwasz for your knowledgeable replies👍. I will investigate your suggested approaches of forwarding all events to an external application using a SocketAppender. 👍