Instrument the application

[severo]

Measure the response time, status code, RAM usage, etc. to be able to take decision (see #1). Also statistics about the most common requests (endpoint, dataset, parameters)

[julien-c]

also probably disk usage as datasets is probably just filling its cache for you forever on disk

[severo]

Yes. The disk usage for datasets cache should be low, since we use streaming (or we error): we never download the data itself. But it's better to monitor than to assume.

[severo]

Note that issues like huggingface/datasets#2859 can affect the CPU and memory usage (even block the server completely)

[severo]

Two solutions proposed by the infra team:

https://huggingface.slack.com/archives/CTKK32GE8/p1630588820038000

---

Morgan:

We have been using Prometheus with success to report this kind of metrics on the Inference API.
Some pointers, if it can help:

• https://prometheus.io/docs/concepts/metric_types/#histogram
• https://prometheus.io/docs/concepts/metric_types/#counter

Philip:

You could also use directly cloudwatch with either the:

• ec2 cloudwatch agent
• or watchtower a python tool, where you use the normal logger and log directly to cloudwatch

AFAIK most of our production service are using prometheus + grafana sofar, with ELK as additional informations. So you could directly log into the existing system for this already

[severo]

See how prometheus is used in other projects:

• https://github.com/huggingface/api-inference/blob/32d4513fb13577d508d06f1f9d0c9dcd17bcad59/master/app/metrics.py
• https://github.com/huggingface/autonlp-backend/search?q=prometheus
• https://github.com/huggingface/streamlit/commit/c77cf30c1e1e319f252f181cb4a1408c7f1f84ec

As we rely on Starlette, we might want to try https://github.com/perdy/starlette-prometheus

[severo]

Also: alert when 500 errors occur. See #21 (comment)

[severo]

Best practices on what to monitor: https://prometheus.io/docs/practices/instrumentation

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[severo]

Already in place. Let's improve when needed