Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opt-in config and logic for deferral of instrumentation to only WSGI worker processes #243

Merged
merged 6 commits into from
Aug 30, 2024

Conversation

srprash
Copy link
Contributor

@srprash srprash commented Aug 28, 2024

Issue:

OTel Python has issues where the SDK is unable to report metrics for applications using a fork process model WSGI server.
This affects ADOT when it tries to generate the OTel or Application Signals metrics.

A solution to this is to re-initialize the SDK in the worker processes after the process forking as happened. A small caveat is that if the SDK has been initialized in the master process, the worker process SDK won't work because Tracer/Meter providers can be set globally only once. So to circumvent this, we need to skip initializing the SDK in the master process and only do so in the worker processes.

Description of changes:

  • Introducing an opt-in configuration environment variable OTEL_AWS_PYTHON_DEFER_TO_WORKERS_ENABLED to enable if they are using a WSGI (or a fork process model) server and want the ADOT SDK to defer auto-instrumentation to worker processes.
  • Whenever the ADOT SDK auto-instrumentation is loaded (either via the sitecustomize.py file or the opentelemetry-instrument command), the SDK will check if the above configuration is enabled and if the current process is the master process, and will skip the instrumentation.
  • The way we determine if the current process is master or worker is by using an internal marker environment variable IS_WSGI_MASTER_PROCESS_ALREADY_SEEN. The first time the ADOT SDK sees a python process, this env var is not set and it will know this should be a WSGI master process. We then set the env var and when a new worker process forks, the master environment is copied to it (and so the env var). So when the ADOT SDK checks this env var again (in worker) it finds that the env var was already set to true in the master.

Testing:

  • Unit tests covering the functionalities bases on different configurations of the OTEL_AWS_PYTHON_DEFER_TO_WORKERS_ENABLED and IS_WSGI_MASTER_PROCESS_ALREADY_SEEN variables.
  • Manual test using a sample application. Since this is an opt-in configuration (a 2-way door), testing manually gives us a fair bit of confidence.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@srprash srprash marked this pull request as ready for review August 29, 2024 22:25
@srprash srprash requested a review from a team as a code owner August 29, 2024 22:25
@srprash srprash changed the title add check to skip instrumentation of wsgi master process Opt-in config and logic for deferral of instrumentation to only WSGI worker processes Aug 29, 2024
Copy link
Member

@mxiamxia mxiamxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@srprash srprash merged commit 90f7fa0 into main Aug 30, 2024
11 checks passed
@srprash srprash deleted the wsgi_worker branch August 30, 2024 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants