Issues with Python AutoInstrumentation

[henrikrexed]

Hi,

I'm currently building content to promote the Operator.
Side of the content I'm building a tutorial showing how the operator injec the auto instrumentation library.
For the moment the nodejs, java services are very well instrumented , but I'm getting errors with my python services.

Here is the log generated on the container :

Failed to load configured propagators
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Instrumenting of requests failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2465, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/__init__.py", line 67, in <module>
    from opentelemetry.propagate import inject
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 117, in initialize
    _load_instrumentors(distro)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 88, in _load_instrumentors
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2465, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/__init__.py", line 67, in <module>
    from opentelemetry.propagate import inject
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
{"timestamp": 1643390481.8418608, "severity": "INFO", "name": "emailservice-server", "message": "starting the email service in dummy mode."}
Failed to load configured propagators
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Traceback (most recent call last):
  File "email_server.py", line 144, in <module>
    start(dummy_mode = True)
  File "email_server.py", line 119, in start
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/__init__.py", line 166, in server
    server_interceptor(tracer_provider=tracer_provider)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/__init__.py", line 252, in server_interceptor
    from . import _server
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/_server.py", line 31, in <module>
    from opentelemetry.propagate import extract
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration

How can i resolve this?

[jpkrohling]

cc @anuraaga, @pavolloffay

[pavolloffay]

@henrikrexed were you able to resolve this? If yes could you please share what was the problem and if no would it be possible to share a docker image with the reproducer?

I am not working on the python ecosystem but @anuraaga could perhaps help here.

[henrikrexed]

Hi,
No, I didn't find a solution to my issue.

here is the repo related to the episode :
https://github.com/isItObservable/OpenTelemetryOperator
The tutorial is not finalized, so don't pay attention to the readme.md.

The image generating the issue are :
hrexed/recommendationservice-otel:0.1
and hrexed/emailservice-otel:0.1
Here is the link to the deployment file :
https://github.com/isItObservable/OpenTelemetryOperator/tree/master/hipster-shop-nootel

Best regards,

[pavolloffay]

Are the images just upstream hipstershop source code?

[henrikrexed]

No i have created a specific version with for a dedicated content explaining how to instrument your code using openTelemetry.
For the content on the operator I have created a "no otel" version that excludes all the opentelemetry , opensensus, opentracing libraries.

Here is the link to the repo :
https://github.com/isItObservable/OpenTelemetry-Instrumentation/tree/mixotel_nonotel/src

[mat-rumian]

Hey @henrikrexed ,
I'm in the middle of testing of OT-Operator capabilities. I think you've faced an issue with default instrumentation yamls :) I mean they are not accurate. Probably you have configured in your Instrumentation CR propagators which are not supported by Python instrumentation. What I found is OT Operator Python instrumentation supports only tracecontext and baggage. All other propagator packages are not installed. If you cleanup your propagators list then your issue should be solved. List of supported instrumentations and propagators is here: https://github.com/open-telemetry/opentelemetry-operator/blob/v0.42.0/autoinstrumentation/python/requirements.txt

[anuraaga]

Thanks @mat-rumian! I think it makes sense to add the other propagators to the instrumentation image too.

[henrikrexed]

Hi, I made the changes on my various deployment files and it works perfectly.
Thanks for your support.
If we need to define a specific propagator on a specific language, does it make sense to specify specific propagators or resources directly at the nodejs, java, or python level ( in the instrumentation CRD)? it would avoid defining several instrumentation objects in the namespace. . what do you think?