FastApi Opentelemetry throwing concurrent.futures._base.CancelledError

[nilansaha]

For Kubernetes I have implemented healthchecks in FastAPI like this

@router.get("/health/liveness", tags=["health_checks"])
async def liveness():
    if check_liveness():
        result = {"status": "UP"}
    else:
        result = {"result": "DOWN"}
    return result

However, opentelemetry always gives errors in these spans. Not sure why. This is the error I get in Datadog Tracing

CancelledError
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/opentelemetry/trace/__init__.py", line 522, in use_span
yield span
File "/usr/local/lib/python3.7/site-packages/opentelemetry/sdk/trace/__init__.py", line 879, in start_as_current_span
yield span_context
File "/usr/local/lib/python3.7/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 213, in wrapped_receive
message = await receive()
File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 526, in receive
await self.message_event.wait()
File "/usr/local/lib/python3.7/asyncio/locks.py", line 293, in wait
await fut
concurrent.futures._base.CancelledError

And this happens for all the endpoints. Not sure whats up. Any help is appreciated.

[thaodt]

hi @nilansaha did u use auto-intrumentation with FastAPI? Did it work? If yes, could you please share your packages version?

[nilansaha]

hi @nilansaha did u use auto-intrumentation with FastAPI? Did it work? If yes, could you please share your packages version?

This is my requirements.txt. Let me know if you can't get it to work. Happy to help. Its so interesting that something like opentelemetry don't update their docs properly and half the things that seem to work does not work. It was painful to make it work

requests==2.25.1
uvicorn==0.13.4
fastapi==0.63.0
opentelemetry-api==1.1.0
opentelemetry-sdk==1.1.0
opentelemetry-distro==0.20b0
opentelemetry-exporter-otlp==1.1.0
opentelemetry-instrumentation==0.20b0
prometheus_fastapi_instrumentator==5.7.1
opentelemetry-instrumentation-fastapi==0.20b0

[thaodt]

hi @nilansaha did u use auto-intrumentation with FastAPI? Did it work? If yes, could you please share your packages version?

This is my requirements.txt. Let me know if you can't get it to work. Happy to help. Its so interesting that something like opentelemetry don't update their docs properly and half the things that seem to work does not work. It was painful to make it work

requests==2.25.1
uvicorn==0.13.4
fastapi==0.63.0
opentelemetry-api==1.1.0
opentelemetry-sdk==1.1.0
opentelemetry-distro==0.20b0
opentelemetry-exporter-otlp==1.1.0
opentelemetry-instrumentation==0.20b0
prometheus_fastapi_instrumentator==5.7.1
opentelemetry-instrumentation-fastapi==0.20b0

Hi @nilansaha I checked with the same required packages (opentelemetry), but it didnt show up my service and my apis in Jaeger dropdown-list.

Maybe you can check my code here.
For vars in settings file, Jaeger is also made from docker container. Their values are:

JAEGER_AGENT_HOST=jaeger   # my docker service name is jaeger
JAEGER_AGENT_PORT=6831

And this is the result:

Could you please help?

[nilansaha]

@thaodt Please share your dockerfile as well

[thaodt]

@thaodt Please share your dockerfile as well

here it is. Please review it. If possible, could you share ur project?

[cdvv7788]

https://github.com/cdvv7788/fastapi_instrumentation_bug I created this repository to reproduce the bug.
Basically you need to move the instrumentation before all of the other middlewares so it is evaluated last (it seems fastapi evaluates middlewares in reverse order). For some reason, the asgi instrumentation is creating additional traces when adding middlewares, which causes this issue to occur.
I don't have much context to know exactly what is happening, but you can reproduce it now.

[github-actions]

This issue was marked stale due to lack of activity. It will be closed in 30 days.

[cdvv7788]

@nilansaha @thaodt This issue is still active. Any ideas to get it solved?

[blueswen]

I think CancelledError exception is a bug in asyncio module of python 3.7. If you using higher version than 3.7 CancelledError problem will be solved. This bug is fix in python 3.8 with this commit, where they use another try to deal with the CancelledError(line 191 of locks.py).

I have the same bug when using fastAPI which has another middleware with opentelemetry manual instrumentation in python 3.7.13. But when I change to python 3.8.13 and python 3.9.12, everything is fine.