-
Notifications
You must be signed in to change notification settings - Fork 581
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to use auto-instrumentation for Uvicorn with multiple worker processes #385
Comments
If you are using Gunicorn as your process manager this should help you https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html. |
This example is not using Gunicorn to run the web server, but if I understand correctly, the pre-fork web server model is the same for Uvicorn. If that's the case, it would be just a matter of finding the post-fork hook in Uvicorn, and applying a similar solution to the ones provided for Gunicorn and uWSGI. The main question here then becomes: is auto-instrumentation possible in the pre-fork web server model? (at least, when Similar to what |
cc @owais, who implemented open-telemetry/opentelemetry-python#1036 |
With Gunicorn, I started receiving spans with the following post_fork hook:
Previously running It's important to note that importing the |
With Uvicorn, it seems to be more complex, as it lacks the required hooks. I started receiving spans after manually editing the
However, if an easy way is found to auto-instrument Gunicorn, IMHO there's no need to work on barebones Uvicorn, but instead to use the Uvicorn worker class for Gunicorn. |
I haven't tried or read the links shared above in detail but isn't the issue about batch span processor and not about instrumentation? Can you try using simple span processor and see if that works to clarify if we are talkig about the same problem? |
This issue was marked stale due to lack of activity. It will be closed in 30 days. |
Closed as inactive. Feel free to reopen if this issue needs resolving. |
Should this be reopened, as Presumably it'd be similar to whatever #676 does, if we had the hook (or monkeypatched the appropriate code in uvicorn). |
I've experienced this issue, and this should probably be re-opened as it should be possible to autoinstrument uvicorn applications. I can confirm that the post fork for uvicorn fixes the issue, but does require app code changes. I did this with a middleware
The fix I actually went with was to move from Demo showing that
|
Please try running with latest rc2 version and let us know if it changes anything. |
@srikanthccv sure, here's a POC using rc2 which demonstrates the problem: https://github.com/bilbof/fastapi-opentel-demo/tree/main/demos/autoinstrument
pip freeze:
AFAIC 0.32b0 is equivalent to 1.12.0rc2, but let me know if I'm wrong and there's a more recent version I can try. |
The problem is caused by Uvicorn using multiprocessing.spawn rather than multiprocessing.fork (which uses os.fork), so the OTel post fork hook isn't running. It'd be great if we could fix this here rather than at uvicorn, since others are experiencing the same problem with multiprocessing (e.g. #2185). Would be interested in your thoughts on how to resolve this. Also, is the fork process model doc still accurate? The workaround in this case is to use gunicorn to run uvicorn apps (which is recommended by Uvicorn) which I've confirmed has worked (see my demo). So issue #2038 could be closed. |
@srikanthccv Can this be resolved soon? We need some uvicorn features that gunicorn doesn't support properly like websockets |
The fork process model documentation is not accurate. I left my comments here open-telemetry/opentelemetry-python#2767 (comment) and here open-telemetry/opentelemetry-python#3307. The logs and traces SDK should always work regardless of the application server used (gunicorn, uvicorn, uwsgi etc...), but the metrics SDK won't work unless there is a hook. I don't know of any fix that can be done in SDK to make it work with Uvicron (or any application server). I guess any threaded application that uses Uvicorn has a high chance of deadlock if there is no way to sanitize the locks&state. I believe upstream should support hooks. |
I am also experiencing the same issue with auto instrumentation and |
Any updates here? |
Describe your environment
Using the following Dockerfile to simplify testing:
Save the file as
Dockerfile
, and build the Docker image it using the following command in the same folder:Steps to reproduce
The Uvicorn server only has one registered endpoint at the root path. Running the server, and then accessing the endpoint (in this example, at http://localhost:5000/) will display any collected spans in the terminal.
Running the Uvicorn server with multiple workers disabled, and OpenTelemetry auto-instrumentation:
Running the Uvicorn server with multiple workers enabled, and OpenTelemetry auto-instrumentation:
What is the expected behavior?
Both modes (single-worker, and multi-worker) must correctly auto-instrument the application, and display collected spans.
What is the actual behavior?
Only the single-worker command displays collected spans. Multi-worker Uvicorn does not display any spans.
Additional context
Uvicorn uses
multiprocessing
to spawn workers. (https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/main.py#L381-L384, and https://github.com/encode/uvicorn/blob/bf1c64e2c1/uvicorn/subprocess.py#L38).Could be related to the same issue that is blocking Gunicorn from working (#171).
Submitting the issue here, as apparently the auto-instrumentation code will be moved to this repository, based on open-telemetry/opentelemetry-python#1532.
The text was updated successfully, but these errors were encountered: