Documentation about using existing Grafana instance. #31667
Conversation
|
This is an amazing PR! Thank you again @tbukic |
| Using an existing Grafana instance | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| When you want to use existing Grafana instance, before starting your Ray cluster you will need to setup environment variable `RAY_GRAFANA_HOST` with an URL of your Grafana. After starting Ray, you can find Grafana dashboard json at `/tmp/ray/session_latest/metrics/grafana/dashboards/default_grafana_dashboard.json`. `Import<https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard>`_ this json to your Grafana. |
There was a problem hiding this comment.
``Importhttps://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard_
Can you tell me a bit about what this exactly means? the URL seems to be linked to the current documentation
There was a problem hiding this comment.
Json at /tmp/ray/session_latest/metrics/grafana/dashboards/default_grafana_dashboard.json is encoding Grafana dashboard. When using existing Grafana, user needs to manually import that json. It is pretty straightforward, but Grafana documentation is showing how it is done, step-by-step. I am going to rephrase it.
|
|
||
| When you want to use existing Grafana instance, before starting your Ray cluster you will need to setup environment variable `RAY_GRAFANA_HOST` with an URL of your Grafana. After starting Ray, you can find Grafana dashboard json at `/tmp/ray/session_latest/metrics/grafana/dashboards/default_grafana_dashboard.json`. `Import<https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard>`_ this json to your Grafana. | ||
|
|
||
| If Grafana reports that datasource is not found, you can `add a datasource variable <https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/?pg=graf&plcmt=data-sources-prometheus-btn-1#add-a-data-source-variable>`_ and change all values of `datasource` key in the dashboard json to the name of the variable. For example, if the variable name is `data_source`, all `"datasource"` mappings should be: |
There was a problem hiding this comment.
"key in the dashboard json"
Is this default_grafana_dashboard.json? Can you change this part to
"key in the default_grafana_dashboard.json"?
There was a problem hiding this comment.
I can! I'm also adding the reference on how to change it.
| nginx.ingress.kubernetes.io/configuration-snippet: | | ||
| add_header X-Frame-Options SAMEORIGIN always; | ||
|
|
||
| When both Grafana and Ray cluster are on the same Kubernetes cluster, it is important to set `RAY_GRAFANA_HOST` to the external URL of the Grafana ingress. For successful embedding, `RAY_GRAFANA_HOST` needs to be accessible to both Ray cluster backend and Ray dashboard frontend: backend does the healthchecks and frontend embeds the dashboard. |
There was a problem hiding this comment.
Can you also add the difference between backend / frontend? E.g.,
backend (ray cluster) and frontend (machines that accesses the dashboard URL)
There was a problem hiding this comment.
Thank you. I've extended on this as well.
Signed-off-by: Bukic Tomislav, AVL <tomislav.bukic@avl.com>
Signed-off-by: Bukic Tomislav, AVL <tomislav.bukic@avl.com>
|
I need an approval from the doc owners! I have asked for the approval, so it will be approved soon! |
Signed-off-by: Bukic Tomislav, AVL tomislav.bukic@avl.com
Added some knowledge about using existing Grafana instances to the monitoring documentation as suggested on Slack channel.
Fixed version of PR #31633
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.