Gracefully handle SIGPIPE

[rossmcdonald]

Earlier versions of journald would send a SIGPIPE to any systemd processes when journald was restarted. So far we have bypassed this issue by redirecting STDOUT and STDERR to the traditional log paths, however in case users opt to use journald it would make sense to handle SIGPIPEs in a graceful manner to ensure the service doesn't crash when journald is restarted.

A similar fix was applied to Docker here: jwhonce/docker@b7f60f7

[nathanielc]

As per the linked patch, it looks like we just need to register a handler to ignore the SIGPIPE and that's it. Correct?

[nathanielc]

I have been reading up on this bug, here is my synposis and my conclusions

systemd version 218 or earlier has a bug where restarting journald causes SIGPIPE to be sent to the process since it can no longer write to STDOUT/STDERR. This means that the process has lost the ability to write logs to journald. In short, for these old versions of systemd restarting journald would cause the process to loose the ability to write anything to STDOUT/STDERR.

Release notes for Go 1.6

On Unix-like systems, when a write to os.Stdout or os.Stderr (more precisely, an os.File opened for file descriptor 1 or 2) fails due to a broken pipe error, the program will raise a SIGPIPE signal. By default this will cause the program to exit; this may be changed by calling the os/signal Notify function for syscall.SIGPIPE. A write to a broken pipe on a file descriptor other 1 or 2 will simply return syscall.EPIPE (possibly wrapped in os.PathError and/or os.SyscallError) to the caller. The old behavior of raising an uncatchable SIGPIPE signal after 10 consecutive writes to a broken pipe no longer occurs.

Now with Go 1.6 (the version Kapacitor uses), SIGPIPE is only sent to the process if a failure occurs on fd 1 or 2, (aka STDOUT and STDERR). This means that if Kapacitor receives a SIGPIPE its because it can no longer write to STDERR or STDOUT. In that case Kapacitor has lost the ability to log anything, and should shutdown.

If you are not using STDOUT/STDERR for Kapacitor logs you will not get SIGPIPE and this issue is irrelevant.

From the os/signal package:

If the program has called Notify to receive SIGPIPE signals, the file descriptor number does not matter. The SIGPIPE signal will be delivered to the Notify channel, and the write will fail with an EPIPE error.

So registering any SIGPIPE handler means getting SIGPIPE for all failed writes, i.e. if a UDF process dies. So we should leave the default behavior in place and call it good.

[nathanielc]

Nothing to do, closing.