-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gracefully handle SIGPIPE #7040
Comments
I concur with @nathanielc's conclusion here for Kapacitor: influxdata/kapacitor#737 (comment) @rossmcdonald I think the proper answer is to upgrade systemd since it seems to be their problem. Unfortunately, we can't modify Go to notify us of a SIGPIPE on only fd 1 or 2 so that we can exit gracefully with an error exit code so systemd causes us to restart and I don't think ignoring SIGPIPE is correct. If this is absolutely necessary though, we can still just ignore the signal and warn people that InfluxDB will not stop if the file descriptor dies. Either way, there's going to need to be a note in the documentation about this issue with a caveat. |
I'm actually seeing this issue on both systemd 208 and 219, so this might be something we need to investigate further. Here is the output from 219:
Depending on how we want to handle this, I think we can either:
I'm thinking the third option is the cleanest, as we're not changing any other behavior apart from making SIGPIPE an exit status. |
I would favor the third option, but for the second option, I don't think we should encourage people to use SIGKILL. It's generally very bad to use that signal. I don't think it would cause any corruption, but it's still not something we want to encourage people to use. |
Oops, sorry, I meant SIGTERM. Agreed that people shouldn't be using SIGKILL on a normal basis. 😄 |
So what is the conclusion? Add cc @nathanielc |
Well it turns out the |
I think |
Which versions are affected by this issue? I just tried it with systemd 229 and no issue. Restarting journald did not send a SIGPIPE and logs are still arriving in the journal after the restart. Why can't we instruct uses with broken versions of systemd to add http://askubuntu.com/questions/659267/how-do-i-override-or-configure-systemd-services Seems like a simple solution that doesn't risk creating issues for our wider user base. |
So far I've only tested with versions 208 and 219, which both exhibited this issue. We can certainly add some documentation around overriding the default unit file if using below a specific systemd version, I was just hoping to find a clean out-of-the-box fix we could use that would have minimal/no impact to current and outdated systemd users alike. Since this isn't as simple of a fix as I was thinking it might be, we might just want to move this over as a docs issue. |
@rossmcdonald @jsternberg @nathanielc Should we close this an re-open as a docs issue? |
Moving this to a doc issue here: influxdata/docs.influxdata.com-ARCHIVE#548 We can create a separate issue if we want to change the default restart behavior when running on systemd. |
But you never actually changed the |
Earlier versions of journald would send a SIGPIPE to any systemd processes when journald was restarted. So far we have bypassed this issue by redirecting STDOUT and STDERR to the traditional log paths, however since we're no longer doing this it would make sense to handle SIGPIPEs in a graceful manner to ensure the service doesn't crash when journald is restarted.
A similar fix was applied to Docker here: jwhonce/docker@b7f60f7
The text was updated successfully, but these errors were encountered: