Gunicorn in docker sends SIGKILL to workers when SIGHUP sent to master process

[antonpetrov145]

Hello all, I have flask-gunicorn api running in docker but the issue is this:
when sending SIGHUP to the master process, in docker it is 1, workers get SIGKILL signals - not gracefully stopped.
this is my config in docker
gunicorn version - 23.0.0

import multiprocessing

bind = "0.0.0.0:5001"
workers = multiprocessing.cpu_count() * 2
pidfile = "api.pid"
errorlog = "/logs/gunicorn-error.log"
keepalive = 10
max_requests = 500
max_requests_jitter = 30
backlog = 800
timeout = 4000
graceful_timeout = 20
capture_output = True
wsgi_app = "wsgi:app"

logs show

[2024-08-14 07:13:07 +0000] [1] [INFO] Starting gunicorn 23.0.0
[2024-08-14 07:13:07 +0000] [1] [INFO] Listening at: http://0.0.0.0:5001 (1)
[2024-08-14 07:13:07 +0000] [1] [INFO] Using worker: sync
[2024-08-14 07:13:07 +0000] [7] [INFO] Booting worker with pid: 7
[2024-08-14 07:13:07 +0000] [8] [INFO] Booting worker with pid: 8
[2024-08-14 07:13:07 +0000] [9] [INFO] Booting worker with pid: 9
[2024-08-14 07:13:07 +0000] [10] [INFO] Booting worker with pid: 10
[2024-08-14 07:14:01 +0000] [1] [INFO] Handling signal: hup
[2024-08-14 07:14:01 +0000] [1] [INFO] Hang up: Master
[2024-08-14 07:14:01 +0000] [28] [INFO] Booting worker with pid: 28
[2024-08-14 07:14:01 +0000] [29] [INFO] Booting worker with pid: 29
[2024-08-14 07:14:01 +0000] [30] [INFO] Booting worker with pid: 30
[2024-08-14 07:14:01 +0000] [9] [INFO] Worker exiting (pid: 9)
[2024-08-14 07:14:01 +0000] [10] [INFO] Worker exiting (pid: 10)
[2024-08-14 07:14:01 +0000] [7] [INFO] Worker exiting (pid: 7)
[2024-08-14 07:14:01 +0000] [8] [INFO] Worker exiting (pid: 8)
[2024-08-14 07:14:01 +0000] [31] [INFO] Booting worker with pid: 31
[2024-08-14 07:14:01 +0000] [1] [ERROR] Worker (pid:8) was sent SIGTERM!
[2024-08-14 07:14:01 +0000] [1] [ERROR] Worker (pid:7) was sent SIGTERM!

gunicorn version - 20.0.4 all works as expected

[2024-08-14 07:17:08 +0000] [1] [INFO] Handling signal: hup
[2024-08-14 07:17:08 +0000] [1] [INFO] Hang up: Master
[2024-08-14 07:17:08 +0000] [120] [INFO] Booting worker with pid: 120
[2024-08-14 07:17:08 +0000] [121] [INFO] Booting worker with pid: 121
[2024-08-14 07:17:08 +0000] [122] [INFO] Booting worker with pid: 122
[2024-08-14 07:17:08 +0000] [123] [INFO] Booting worker with pid: 123
[2024-08-14 07:17:08 +0000] [71] [INFO] Worker exiting (pid: 71)
[2024-08-14 07:17:08 +0000] [67] [INFO] Worker exiting (pid: 67)
[2024-08-14 07:17:08 +0000] [76] [INFO] Worker exiting (pid: 76)
[2024-08-14 07:17:08 +0000] [64] [INFO] Worker exiting (pid: 64)

what am I doing wrong?

[pajod]

Does anything besides the SIGTERM error-severity log make you believe that worker shutdown was abnormal?

• likely dupe: Error instead of warning in logs when sending SIGTERM since v21 #3050

[antonpetrov145]

@pajod good question, and it seems this is duplicate to the linked issue.

Anyway I tried to reload the workers and despite the SIGTERM - workers did consume the requests correctly, but logs get huge with messages like these

--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1164, in emit
    self.flush()
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1144, in flush
    self.stream.flush()
RuntimeError: reentrant call inside <_io.BufferedWriter name='/logs/gunicorn-error.log'>

Call stack:
  File "/usr/local/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/wsgiapp.py", line 66, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]", prog=prog).run()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/base.py", line 235, in run
    super().run()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/app/base.py", line 71, in run
    Arbiter(self).run()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 208, in run
    self.sleep()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 359, in sleep
    ready = select.select([self.PIPE[0]], [], [], 1.0)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 241, in handle_chld
    self.reap_workers()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 553, in reap_workers
    self.log.error(msg)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/glogging.py", line 271, in error
    self.error_log.error(msg, *args, **kwargs)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1568, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1684, in _log
    self.handle(record)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1700, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1028, in handle
    self.emit(record)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1280, in emit
    StreamHandler.emit(self, record)
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1164, in emit
    self.flush()
  File "/usr/local/lib/python3.12/logging/__init__.py", line 1144, in flush
    self.stream.flush()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 241, in handle_chld
    self.reap_workers()
  File "/usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py", line 553, in reap_workers
    self.log.error(msg)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/glogging.py", line 271, in error
    self.error_log.error(msg, *args, **kwargs)
Message: 'Worker (pid:35) was sent SIGTERM!'
Arguments: ()