kubeflow components metrics not merged by istio

[axel7083]

Description

I am trying to collect all metrics from the different Kubeflow components and particularly the notebook controller using Prometheus.

Environment

The configuration is the one by default.

• The cluster-local-gateway has the 15020 port open exposing the /metrics and /stats/prometheus¹.
• The istio configmap in the istio-system namespace has enablePrometheusMerge: true.

The annotations are properly overwritten by istio for the notebook-controller-deployment, here is an example:

kubectl get pod -n kubeflow notebook-controller-deployment-... -o jsonpath='{.metadata.annotations}' give

{
    ...
    "prometheus.io/path": "/stats/prometheus",
    "prometheus.io/port": "15020",
    "prometheus.io/scrape": "true",
    "sidecar.istio.io/status": "...."
}

Calling the /stats/prometheus on notebook service will result in the same metrics from the cluster local gateway.

Reproduce

• Forward notebook-controller-service: kubectl port-forward -n kubeflow svc/notebook-controller-service 8080:8080
• Forward cluster-local-gateway service: kubectl port-forward -n istio-system svc/cluster-local-gateway 15020:15020

Go on localhost:8080/metrics and compare with localhost:15020/stats/prometheus, they are no corresponding metrics.

Footnotes

1. Istio documentation ↩

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.