Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] implement last_activity collection via prometheus metrics #229

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

minrk
Copy link
Member

@minrk minrk commented Jan 17, 2024

two modes:

  • compute via traefik API. Only valid if one replica
  • compute via prometheus. Required if more than one replica

TODO:

  • implement last_activity computation via traefik API (requires implementing simple prometheus collect & sum)
  • implement last_activity computation via prometheus query
  • decide what to do about websocket activity
  • resolve configuration situations for:
    • enable metrics without last_activity
    • enable last_activity with only single endpoint
    • enable last_activity via prometheus (incl auth?)
  • tests
  • docs

As it currently stands, we have the following configuration:

c.TraefikProxy.enable_last_activity = True

enables single-replica last_activity via the existing traefik API endpoint. _collect_last_activity_api fetches the prometheus metrics endpoint of traefik and parses the relevant metric lines, computing the equivalent of

sum(traefik_router_requests_bytes_total) by (router) + 
sum(traefik_router_responses_bytes_total) by (router)

"activity" is registered as an increase in these values since the last collection (in memory, reset on first collection)

c.TraefikProxy.enable_last_activity = True
c.TraefikProxy.last_activity_prometheus_url = "http://localhost:9090"

requires a prometheus instance configured to collect from traefik, and uses the query:

sum(increase(traefik_router_requests_bytes_total{interval})) by (router) + 
sum(increase(traefik_router_responses_bytes_total{interval})) by (router)

where interval is the number of seconds since the last request. This should produce the same results as above, but handle aggregation across multiple replicas and doesn't require comparison with previous results (increase handles that).

Notably, it seems that websocket messages do not produce measured activity on any traefik metrics: traefik/traefik#10358

Fortunately (or not), JupyterLab still sends tons of API requests even while it is idle, so relying on websocket messages for traffic in 2024 is less critical than it was in the much less resource-intensive days of the classic notebook. But it's still something to be aware of, because active websocket messages are how activity is typically tracked in CHP (e.g. output being produced or execute requests being sent).

Another metric to consider is the open_connections metric, which allows us to consider any open websocket connection to be 'currently active'. This means any open tab (even an idle one) would prevent culling. Sometimes that's good, sometimes it's not, which is why Jupyter Server's internal culler has config to select whether this is considered or not.

closes #151

two modes:

- compute via traefik API. Only valid if one replica
- compute via prometheus. Required if more than one replica
@minrk minrk added the new new features label Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

explore activity tracking via metrics
1 participant