From 83ed23712fab545fb42b648fa0ea96da9e065919 Mon Sep 17 00:00:00 2001
From: Suraj Deshmukh <suraj@kinvolk.io>
Date: Thu, 4 Jun 2020 17:52:27 +0530
Subject: [PATCH 1/2] docs: How to add custom monitoring resources

This doc explains users how to add custom monitoring resources like
Grafana dashboards, Prometheus scrape targets using ServiceMonitors,
alerts using PrometheusRules.

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>
---
 .../monitoring-with-prometheus-operator.md    | 104 ++++++++++++++++--
 1 file changed, 95 insertions(+), 9 deletions(-)

diff --git a/docs/how-to-guides/monitoring-with-prometheus-operator.md b/docs/how-to-guides/monitoring-with-prometheus-operator.md
index bcbf41415..d8e7c322a 100644
--- a/docs/how-to-guides/monitoring-with-prometheus-operator.md
+++ b/docs/how-to-guides/monitoring-with-prometheus-operator.md
@@ -4,9 +4,9 @@
 
 * [Introduction](#introduction)
 * [Prerequisites](#prerequisites)
-* [Deploy Prometheus Operator](#deploy-prometheus-operator)
-  * [Configure Prometheus Operator](#configure-prometheus-operator)
-  * [Install Prometheus Operator](#install-prometheus-operator)
+* [Steps: Deploy Prometheus Operator](#steps-deploy-prometheus-operator)
+  * [Step 1: Configure Prometheus Operator](#step-1-configure-prometheus-operator)
+  * [Step 2: Install Prometheus Operator](#step-2-install-prometheus-operator)
 * [Accessing Prometheus, Alertmanager and Grafana](#accessing-prometheus-operator-sub-components)
   * [Accessing Prometheus](#accessing-prometheus)
     * [Using port forward](#using-port-forward)
@@ -15,6 +15,13 @@
   * [Accessing Grafana](#accessing-grafana)
     * [Using port forward](#using-port-forward-1)
     * [Using Ingress](#using-ingress-1)
+* [Add custom Grafana dashboards](#add-custom-grafana-dashboards)
+* [Add new ServiceMonitors](#add-new-service-monitors)
+  * [Default Prometheus operator setting](#default-prometheus-operator-setting)
+  * [Custom Prometheus operator setting](#custom-prometheus-operator-setting)
+* [Add custom alerts for Alertmanager](#add-custom-alerts-for-alertmanager)
+  * [Default Prometheus operator setting](#default-prometheus-operator-setting-1)
+  * [Custom Prometheus operator setting](#custom-prometheus-operator-setting-1)
 * [Additional resources](#additional-resources)
 
 ## Introduction
@@ -26,10 +33,10 @@ This guide provides the steps for deploying a monitoring stack using the `promet
 * A Lokomotive cluster deployed on a supported provider and accessible via `kubectl`.
 
 <!---
-TODO: Once we have tutorials on how to deploy and configure Rook or OpenEBS, point the following to those tutorials.
+TODO: Once we have tutorials on how to deploy and configure OpenEBS, point the following to those tutorials.
 -->
 
-* A storage provider component (`rook` and `rook-ceph`, or `openebs-operator` and `openebs-storage-class`) deployed with a default storage class that can provision volumes for the [PVCs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) created by Alertmanager and Prometheus.
+* A storage provider component ([`rook` and `rook-ceph`](./rook-ceph-storage.md), or `openebs-operator` and `openebs-storage-class`) deployed with a default storage class that can provision volumes for the [PVCs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) created by Alertmanager and Prometheus.
 
 <!---
 TODO: Once we have a tutorial on how to deploy and configure Contour and cert-manager, point the following to that tutorial.
@@ -40,9 +47,9 @@ TODO: Once we have a tutorial on how to deploy and configure Contour and cert-ma
 > * [`cert-manager`](../configuration-reference/components/cert-manager.md)
 
 
-## Deploy Prometheus Operator
+## Steps: Deploy Prometheus Operator
 
-### Configure Prometheus Operator
+### Step 1: Configure Prometheus Operator
 
 Create a file named `monitoring.lokocfg` with the following contents:
 
@@ -50,9 +57,9 @@ Create a file named `monitoring.lokocfg` with the following contents:
 component "prometheus-operator" {}
 ```
 
-For information about all the available configuration options for the `prometheus-operator` component, visit the component's [configuration reference](../configuration-reference/components/prometheus-operator.md).
+For information about all the available configuration options for the `prometheus-operator` component, visit the component's [configuration reference](../configuration-reference/components/prometheus-operator.md). If you would like to add custom Alerts and Grafana dashboards then look at the section ["Add custom Grafana dashboards"](#add-custom-grafana-dashboards) and subsequent sections.
 
-### Install Prometheus Operator
+### Step 2: Install Prometheus Operator
 
 Execute the following command to deploy the `prometheus-operator` component:
 
@@ -152,6 +159,85 @@ kubectl -n monitoring get secret prometheus-operator-grafana -o jsonpath='{.data
 
 Open the following URL: `https://grafana.<cluster name>.<DNS zone>`. Enter the username `admin` and the password obtained from the previous step.
 
+## Add custom Grafana dashboards
+
+Create a ConfigMap with keys as the dashboard file names and values as JSON dashboard. See the following example:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboards
+  namespace: myapp
+  labels:
+    grafana_dashboard: "true"
+data:
+  grafana-dashboard1.json: |
+    {
+      "annotations": {
+[REDACTED]
+```
+
+Add the label `grafana_dashboard: "true"` so that grafana automatically picks up the dashboards in the ConfigMaps across the cluster.
+
+This can also be done by using the following two imperative commands:
+
+```bash
+kubectl -n myapp create cm grafana-dashboards \
+  --from-file=grafana-dashboard1.json \
+  --from-file=grafana-dashboard2.json \
+  --dry-run -o yaml | kubectl apply -f -
+
+kubectl -n myapp label cm grafana-dashboards grafana_dashboard=true
+```
+
+## Add new ServiceMonitors
+
+### Default Prometheus operator setting
+
+Create a ServiceMonitor with the required configuration and make sure to add the following label, so that the prometheus-operator will track it:
+
+```yaml
+metadata:
+  labels:
+    release: prometheus-operator
+```
+
+### Custom Prometheus operator setting
+
+Deploy the prometheus-operator with the following setting, and it watches all ServiceMonitors across the cluster:
+
+```tf
+watch_labeled_service_monitors = "false"
+```
+
+Then there is no need to add any label to ServiceMonitor, at all. Create a ServiceMonitor, and prometheus-operator tracks it.
+
+## Add custom alerts for Alertmanager
+
+### Default Prometheus operator setting
+
+Create a PrometheuRule object with the required configuration and make sure to add the following labels, so that prometheus-operator will track it:
+
+```yaml
+metadata:
+  labels:
+    release: prometheus-operator
+    app: prometheus-operator
+```
+
+### Custom Prometheus operator setting
+
+Deploy the prometheus-operator with the following setting, and it watches all PrometheusRules across the cluster:
+
+```tf
+watch_labeled_prometheus_rules = "false"
+```
+
+Then there is no need to add any label to PrometheusRule, at all. Create a PrometheusRule, and prometheus-operator tracks it.
+
 ## Additional resources
 
 - `prometheus-operator` component [configuration reference](../configuration-reference/components/prometheus-operator.md).
+- ServiceMonitor API docs https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#servicemonitor
+- PrometheusRule API docs https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrule

From c4a88fef4b2f1eb8e8430d7d5596237db11ecc22 Mon Sep 17 00:00:00 2001
From: Suraj Deshmukh <suraj@kinvolk.io>
Date: Tue, 4 Aug 2020 17:31:19 +0530
Subject: [PATCH 2/2] prometheus-operator docs: Modify the verbs to simple
 present tense

In the "How to guide" of prometheus-operator this commit changes the
titles of sections from present continuous tense to simle present tense.

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>
---
 .../monitoring-with-prometheus-operator.md     | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/docs/how-to-guides/monitoring-with-prometheus-operator.md b/docs/how-to-guides/monitoring-with-prometheus-operator.md
index d8e7c322a..2bdbe7823 100644
--- a/docs/how-to-guides/monitoring-with-prometheus-operator.md
+++ b/docs/how-to-guides/monitoring-with-prometheus-operator.md
@@ -1,4 +1,4 @@
-# Setting up monitoring on Lokomotive
+# Set up monitoring on Lokomotive
 
 ## Contents
 
@@ -7,12 +7,12 @@
 * [Steps: Deploy Prometheus Operator](#steps-deploy-prometheus-operator)
   * [Step 1: Configure Prometheus Operator](#step-1-configure-prometheus-operator)
   * [Step 2: Install Prometheus Operator](#step-2-install-prometheus-operator)
-* [Accessing Prometheus, Alertmanager and Grafana](#accessing-prometheus-operator-sub-components)
-  * [Accessing Prometheus](#accessing-prometheus)
+* [Access Prometheus, Alertmanager and Grafana](#access-prometheus-alertmanager-and-grafana)
+  * [Access Prometheus](#access-prometheus)
     * [Using port forward](#using-port-forward)
     * [Using Ingress](#using-ingress)
-  * [Accessing Alertmanager](#accessing-alertmanager)
-  * [Accessing Grafana](#accessing-grafana)
+  * [Access Alertmanager](#access-alertmanager)
+  * [Access Grafana](#access-grafana)
     * [Using port forward](#using-port-forward-1)
     * [Using Ingress](#using-ingress-1)
 * [Add custom Grafana dashboards](#add-custom-grafana-dashboards)
@@ -73,9 +73,9 @@ Verify the pods in the `monitoring` namespace are in the `Running` state (this m
 kubectl -n monitoring get pods
 ```
 
-## Accessing Prometheus, Alertmanager and Grafana
+## Access Prometheus, Alertmanager and Grafana
 
-### Accessing Prometheus
+### Access Prometheus
 
 #### Using port forward
 
@@ -107,7 +107,7 @@ component "prometheus-operator" {
 
 Open the following URL: `https://prometheus.<cluster name>.<DNS zone>`.
 
-### Accessing Alertmanager
+### Access Alertmanager
 
 Execute the following command to forward port `9093` locally to the Alertmanager pod:
 
@@ -117,7 +117,7 @@ kubectl -n monitoring port-forward svc/prometheus-operator-alertmanager 9093
 
 Open the following URL: [http://localhost:9093](http://localhost:9093).
 
-### Accessing Grafana
+### Access Grafana
 
 #### Using port forward