Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Relic Instrumentor on FastAPI app fails when both OpenTelemetry and Middleware are setup #1136

Open
csuriano23 opened this issue May 8, 2024 · 2 comments

Comments

@csuriano23
Copy link

Instrumenting a FastAPI app with New Relic executable (newrelic-admin run-program) causes failures in app endpoints due to a type mismatch on the ASGI middlewares chain

Description
Using this minimal example FastAPI app:

import newrelic.agent
from fastapi import FastAPI
from fastapi.middleware.gzip import GZipMiddleware
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider

from .models.responses import JSONResponse

app = FastAPI(default_response_class=JSONResponse)

app.add_middleware(GZipMiddleware, minimum_size=1000)


@app.get("/healthcheck")
async def healthcheck() -> JSONResponse:
    newrelic.agent.ignore_transaction()
    return JSONResponse({})


# Setup OpenTelemetry
tracer = TracerProvider()
trace.set_tracer_provider(tracer)
FastAPIInstrumentor.instrument_app(app, excluded_urls="/docs,/healthcheck/")

Dependencies versions (excerpt from pyproject.toml):

[tool.poetry.dependencies]
python = "^3.10"
newrelic = "9.9.0"
uvicorn = ">=0.21.1,<1"  # installed 0.29.0
fastapi = ">=0.109.2,<1"  # installed FastAPI 0.110.1 with Starlette 0.37.2
opentelemetry-instrumentation-fastapi = "^0.45b0"
opentelemetry-instrumentation-requests = "^0.45b0"
opentelemetry-instrumentation-pymongo = "^0.45b0"
opentelemetry-instrumentation-redis = "^0.45b0"
opentelemetry-instrumentation-logging = "^0.45b0"
opentelemetry-api = "^1.24.0"
opentelemetry-sdk = "^1.24.0"

Running the app with the command:

newrelic-admin run-program bash -c "uvicorn main:app --host 0.0.0.0 --port 8000"

causes an exception when calling the app endpoints (in the example: "/healthcheck")

The exception stacktrace is:

/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py:-1: RuntimeWarning: coroutine 'middleware_wrapper' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
INFO:     127.0.0.6:59175 - "GET /healthcheck/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/asgi_application.py", line 357, in nr_async_asgi
    return await coro
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/asgi_application.py", line 357, in nr_async_asgi
    return await coro
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/usr/local/lib/python3.10/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/newrelic/hooks/framework_starlette.py", line 112, in middleware_wrapper
    return await FunctionTraceWrapper(wrapped, name=name)(*args, **kwargs)
  File "<string>", line 5, in wrapper
  File "/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 575, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/asgiref/compatibility.py", line 35, in new_application
    return await instance(receive, send)
TypeError: 'coroutine' object is not callable

If I remove:

  • the GZIP middleware, but NOT the OpenTelemetry instrumentor

OR

  • the OpenTelemetry instrumentor, but NOT the GZIP middleware

this is not happening.
The OpenTelemetry instrumentor itself contains an ASGI middleware, so I think something is not working properly in the interceptors chain when both are installed. Anyway the OpenTelemetry instrumentor shouldn't be incompatible with the NewRelic one.

@csuriano23
Copy link
Author

Any update on this?

@vitaliy-sk
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants