Log a warning when a worker was terminated due to a signal #2475
Conversation
This happens for example when being OOM killed. See #2215
|
which signal to who are we talking? Is kubernetes sending a SIGKILL signal? Anyway I am fine with such changes. |
Kill. When you have memory limits set in Kubernetes it can easily happen that a worker is shot down without any kind of logging/warning. Restarting works nicely, it is just that having a log eases the post mortem. |
|
i would suggest to configure kubernetes to send a sigterm instead of a
sigkill signal instead. That way the worker will be terminated gracefully/
Also if you know your application will leak or use a lot of ram over the
time you should limit the number of connections a worker can accept until
it is killing itself.
…On Thu, Dec 17, 2020 at 11:29 AM Armin Berres ***@***.***> wrote:
which signal to who are we talking?
Kill.
When you have set memory limits set in Kubernetes it can easily happen
that a worker is shot down without any kind of logging. Restarting work
nicely, it is just that having a log eases the post mortem.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#2475 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAADRIUTDL5GQJDYBKCVZQLSVHMRTANCNFSM4U5XG23A>
.
|
|
Yeah, that's an ongoing discussion of it should not send a SIGKILL immediately. But not likely to be changed as I understood the issues I was looking at. More often than not when this situation arises it is likely a configuration issue and the requested memory should be increased. Thanks a lot for the merge! |
|
K8s does send TERM for normal shutdown, but OOM killer in Linux always sends KILL, I think. |
Original: b695b49 (benoitc#2475) Reverted: 76f8da2 The warnings are printed later in the main loop, not directly in the signal handler. This should fix the RuntimeError from benoitc#2564.
This happens for example when being OOM killed.
See #2215