reorg into folders, add hotROD dashboard, add basic description to RE…

…ADME

examples/grafana-integration/README.md

@@ -1,14 +1,27 @@
 # Hot R.O.D. - Rides on Demand  - Grafana integration
 
-This is the Hot R.O.D. demo application that consists of the same components as the `examples/hotrod/`, only Grafana and Loki integration is added to this setup, so you can correlate logs and traces in one application.
+This is the Hot R.O.D. demo application that consists of the same components as the `examples/hotrod/`, only Grafana, Loki and Prometheus integration is added to this setup, so you can correlate logs, metrics and traces in one application.
 
-## Running
+## Running via `docker-compose`
 
-### Run everything via `docker-compose`
+### Prerequisites
+
+* Clone the Jaeger repository `git clone https://github.com/jaegertracing/jaeger.git`, then `cd examples/grafana-integration`
+
+* All services will log to Loki via the loki docker logging plugin.
+To install run this on the host:
+
+```bash
+docker plugin install \
+grafana/loki-docker-driver:latest \
+--alias loki \
+--grant-all-permissions
+```
+
+### Deploy the services
+
+* Pull up the services: `docker-compose up` 
 
-#### Prerequisites
-* Download `docker-compose.yml` from https://github.com/jaegertracing/jaeger/blob/master/examples/grafana-integration/docker-compose.yml
-* Download the `datasources.yaml` from the `examples/grafana-integration/` folder
 * All services will log to Loki via the loki docker logging plugin.
 To install run this on the host once:
 
@@ -19,8 +32,35 @@ grafana/loki-docker-driver:latest \
 --grant-all-permissions
 ```
 
-#### Deploy the local setup
-* Run Grafana and Loki integration with Jaeger backend using HotROD demo with `docker-compose -f path-to-yml-file up`
-* Access Grafana UI at http://localhost:3000 and HotROD app at http://localhost:8080
-* Shutdown / cleanup with `docker-compose -f path-to-yml-file down`
+### Access the services
+* HotROD application at http://localhost:8080
+* Access Grafana UI at http://localhost:3000
+
+### Explore with Loki
+
+Currently the most powerful way to correlate application logs with traces can be performed via Grafana's Explore interface.
+
+After setting the datasource to Loki, all the log labels become available, and can be easily filtered using [Loki's LogQL query language](https://grafana.com/docs/loki/latest/logql/).
+
+For example after selecting the compose project, errors can be filtered with the following expression:
+
+```
+{compose-project="grafana-integration"} |= "error"
+```
+
+which will list the redis timeout events.
+
+### HotROD - Metrics and logs overview dashboard
+
+Since the HotROD application can expose its metrics in Prometheus' format, these can be also used during investigation.
+
+This example includes a dashboard, which contains a log panel, which displays the logs for the selected services in real time. These can be also filtered by a search field.
+
+There are also panels to display the ratio/percentage of errors in the current timeframe.
+
+Additionally, there are graphs for each service, visualing the rate of the requests and showing latency percentiles.
+
+### Clean up
+
+To stop the containers use `docker-compose -f path-to-yml-file down`
 

examples/grafana-integration/docker-compose.yaml

@@ -1,11 +1,12 @@
 version: '2'
+
 services:
   grafana:
     image: grafana/grafana:7.1.4
     ports:
       - '3000:3000'
     volumes:
-      - ./datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
+      - ./grafana/provisioning/:/etc/grafana/provisioning/
     logging:
       driver: loki
       options:
@@ -16,6 +17,7 @@ services:
     ports:
       - '3100:3100'
     command: -config.file=/etc/loki/local-config.yaml
+    # send Loki traces to Jaeger
     environment:
       - JAEGER_AGENT_HOST=jaeger
       - JAEGER_AGENT_PORT=6831
@@ -39,9 +41,9 @@ services:
   hotrod:
     image: jaegertracing/example-hotrod:latest
     ports:
-      - "8080:8080"
-      - "8083:8083"
-    command: ["-m", "prometheus","all"]
+      - '8080:8080'
+      - '8083:8083'
+    command: ["-m","prometheus","all"]
     environment:
       - JAEGER_AGENT_HOST=jaeger
       - JAEGER_AGENT_PORT=6831
@@ -51,11 +53,11 @@ services:
         loki-url: 'http://localhost:3100/api/prom/push'
 
   prometheus:
-    image: prom/prometheus:v2.19.0
+    image: prom/prometheus:v2.19.1
     volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
     ports:
-      - 9090:9090
+      - '9090:9090'
     command:
     - --config.file=/etc/prometheus/prometheus.yml
     logging:

examples/grafana-integration/grafana/provisioning/dashboards/dashboard.yml

@@ -0,0 +1,11 @@
+apiVersion: 1
+
+providers:
+- name: 'HotROD'
+  orgId: 1
+  folder: ''
+  type: file
+  disableDeletion: false
+  editable: true
+  options:
+    path: /etc/grafana/provisioning/dashboards