Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added log before waiting on threads #2556

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jafar-atili
Copy link

@jafar-atili jafar-atili commented Aug 22, 2023

Why do we need this log?

We manage a large Python application. once we started using the max-requests we noticed that clients are receiving 502 during the max-request reload.

It is worth mentioning that we have 5x uwsgi processes, each with 70x threads.

Our first step toward resolving this behavior was adding the reload-mercy/worker-reload-mercy to grant our application 20 minutes to finish up the old requests and gracefully shut down the worker.

This still didn't resolve our problem, and here we wanted to understand what is preventing our workers from shutting down gracefully.

In order to investigate it, we wanted to understand what happens inside our application before the reload to understand what is stuck and prevent our application from shutting down gracefully.

we compiled a version of uwsgi with this PR, and once we saw the new log message we knew this is our window (until the mercy-reload timeout) to dump the process stacktrace (we used py-spy) from the host.

@methane
Copy link
Contributor

methane commented Mar 19, 2024

max-requests-delta will help you. Although this is 3.0 only option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants