Opt-in config and logic for deferral of instrumentation to only WSGI worker processes #243
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue:
OTel Python has issues where the SDK is unable to report metrics for applications using a fork process model WSGI server.
This affects ADOT when it tries to generate the OTel or Application Signals metrics.
A solution to this is to re-initialize the SDK in the worker processes after the process forking as happened. A small caveat is that if the SDK has been initialized in the master process, the worker process SDK won't work because Tracer/Meter providers can be set globally only once. So to circumvent this, we need to skip initializing the SDK in the master process and only do so in the worker processes.
Description of changes:
OTEL_AWS_PYTHON_DEFER_TO_WORKERS_ENABLED
to enable if they are using a WSGI (or a fork process model) server and want the ADOT SDK to defer auto-instrumentation to worker processes.sitecustomize.py
file or theopentelemetry-instrument
command), the SDK will check if the above configuration is enabled and if the current process is the master process, and will skip the instrumentation.IS_WSGI_MASTER_PROCESS_ALREADY_SEEN
. The first time the ADOT SDK sees a python process, this env var is not set and it will know this should be a WSGI master process. We then set the env var and when a new worker process forks, the master environment is copied to it (and so the env var). So when the ADOT SDK checks this env var again (in worker) it finds that the env var was already set totrue
in the master.Testing:
OTEL_AWS_PYTHON_DEFER_TO_WORKERS_ENABLED
andIS_WSGI_MASTER_PROCESS_ALREADY_SEEN
variables.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.