BrokenResourceError on metrics endpoint

[mdczaplicki]

They mention here that it's related to MemoryStream middleware, but I'm not sure it's relevant.
fastapi/fastapi#4041

Versions:
PFI: 5.8.2
FastAPI 0.78
Starlette 0.19.1

Any ideas how to fix it?

[circlingthesun]

I've got exactly the same problem. I get this error for a bunch of endpoints but the most problematic one is the health endpoint that's pretty simple:

@router.get("/health")
async def health():
    val = await database.fetch_val("SELECT 1")

    if val == 1:
        return "Ok"

    raise HTTPException(500, "Can't connect to DB")

Here is a bigger stack trace screenshot:

[mdczaplicki]

It happens randomly on all endpoints. Maybe your health endpoint is used the most, therefore it seems most problematic for you.

[circlingthesun]

@mdczaplicki yeah, it happens on all endpoints, but the health endpoint restarts my service when it fails too often, which is a problem. It's also a pretty simple endpoint, so I thought it's worth including for illustrative purposes.

[circlingthesun]

@mdczaplicki I upgraded FastAPI, prometheus-fastapi-instrumentator and uvicorn to the latest versions, and I haven't seen this error in the last hour 🤞

[mdczaplicki]

I did as well. Only FAPI was to be updated, though didn't go away. :/

[mdczaplicki]

@circlingthesun, what other 3rd party packages do you use in your project? Do you still not get the error?

I've realized that our pods are also restarted, because health endpoint fails.

[circlingthesun]

@mdczaplicki I don't use any other middleware (which seems to be the culprit) beside the starlette/fastapi CORS middleware. If you think you can make progress by looking at my dependencies here you go:

alembic==1.8.1
aio_pika==7.2.0
Babel==2.10.1
bandit==1.7.4
beautifulsoup4==4.11.1
base58==2.1.1
billiard==3.6.3.0
celery==4.4.2
flake8==4.0.1
GeoAlchemy2==0.12.2
icalendar==4.1.0
intervaltree==3.1.0
iso3166==2.0.2
Jinja2==3.1.2
lxml==4.8.0
PyJWT==1.7.1
bcrypt==3.2.2
mock==4.0.3
mypy==0.971
pika==1.2.1
Pillow==9.2.0
prometheus-client==0.14.1
types_python_dateutil==2.8.18
pytz==2022.1
types_pytz==2022.1.1
requests==2.25.1
sentry-sdk==1.8.0
simplejson==3.17.2
types_simplejson==3.17.7
SQLAlchemy==1.4.39
SQLAlchemy-Searchable==1.4.1
WeasyPrint==56.1
uvicorn[standard]==0.18.2
fastapi==0.79.0
pydantic==1.9.1
python-multipart==0.0.5
websockets==10.3
databases==0.6.0
asyncpg==0.26.0
cachetools==4.2.1
types-cachetools==5.2.1
email-validator==1.2.1
httpx==0.23.0
watchdog==2.1.7
argh==0.26.2
python_jose[cryptography]==3.3.0
bleach==5.0.0
types-bleach==5.0.3
passlib[bcrypt]==1.7.4
aiobotocore==1.2.2
boto3==1.16.52
asyncache==0.1.1
dacite==1.6.0
locust==2.10.1
watchfiles==0.16.1
prometheus_fastapi_instrumentator==5.8.2
asgiref==3.5.2

[circlingthesun]

@mdczaplicki I forgot I'm also using sentry's middleware here are the relavant lines:

app.add_middleware(SentryAsgiMiddleware)

# Make sure this is last? https://github.com/tiangolo/fastapi/issues/1663
app.add_middleware(
    CORSMiddleware,
    allow_origin_regex="https?://.*",
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Instrumentator().instrument(app).expose(app)

[mdczaplicki]

Hmm, and you're sure that you get rid of BrokenResourceError? We are trying to get rid of the CORSMiddleware from starlette, we have a suspicion, that it's the culprit.

I suppose you have starlette==0.19.1, right?

[circlingthesun]

@mdczaplicki yes, starlette==0.19.1, I don't lock it, I just get whatever version FastAPI depends on. Also just to be complete, I get the CORS middleware via from fastapi.middleware.cors import CORSMiddleware.

[mdczaplicki]

Uuu, thanks for the hint. We use from starlette.middleware.cors import CORSMiddleware. If we'll get rid of this error, by removing starlette's CORSMiddleware - we'll switch to fastapi's

[circlingthesun]

@mdczaplicki its not completely gone, but since upgrading my dependencies on the 26th, there has been exactly 3 occurrences:

[mdczaplicki]

🤣

[circlingthesun]

@mdczaplicki ha ha, yeah, I vaguely recalled that being the case.

[mdczaplicki]

@circlingthesun, hey man. I'm coming back to you - to let you know that the problem is not with this library.

It's related to Sentry middleware. You should update to newest Sentry SDK (1.9.0). In 1.8.0 they've introduced new FastAPI integration method. However, don't rush into updating yet, because this will cause new errors to pop up - much more harsh.

Better wait until this PR is merged: getsentry/sentry-python#1532

New integration configuration: https://docs.sentry.io/platforms/python/guides/fastapi/#configure

You can also downgrade to 1.5.x to get rid of BrokenResourceError.

[circlingthesun]

@mdczaplicki it's funny that without sentry we wouldn't be aware of the problem but also wouldn't have the problem. Thanks for the heads up 👍