Improve Helm chart - Use metric sink approach for Prometheus & genera…

…l refresh (#1260)

build/azure-devops/templates/helm/package-official-chart.yml

@@ -18,4 +18,4 @@ steps:
     mkdir output/
     helm package ${{ parameters.chartName }}/ --version ${{ parameters.chartVersion }} --app-version ${{ parameters.appVersion }} --destination output/
   workingDirectory: charts
-  displayName: 'Package Helm Chart v${{ parameters.chartVersion }} (App version ${{ parameters.appVersion }})'
+  displayName: 'Package Helm Chart v${{ parameters.chartVersion }} (App version ${{ parameters.appVersion }})'

charts/promitor-agent-scraper/Chart.yaml

@@ -1,10 +1,13 @@
-apiVersion: v1
+apiVersion: v2
 version: 0.2.0
 appVersion: 1.0.0
-description: Promitor, bringing Azure Monitor metrics where you need them.
+type: application
 name: promitor-agent-scraper
-icon: https://raw.githubusercontent.com/tomkerkhove/promitor/master/docs/media/logos/promitor.png
-home: https://github.com/tomkerkhove/promitor
+description: Promitor, bringing Azure Monitor metrics where you need them.
+home: https://promitor.io
+sources:
+  - https://github.com/tomkerkhove/promitor
 maintainers:
   - name: Tom Kerkhove
-    url: https://github.com/tomkerkhove
+    url: https://github.com/tomkerkhove
+icon: https://raw.githubusercontent.com/tomkerkhove/promitor/master/docs/media/logos/promitor.png

charts/promitor-agent-scraper/README.md

@@ -1,14 +1,15 @@
 # Promitor
 
-[Promitor](https://promitor.io/) is an Azure Monitor scraper for Prometheus providing
-a scraping endpoint for Prometheus that provides a configured subset of Azure
-Monitor metrics.
+[Promitor](https://promitor.io/) is an Azure Monitor scraper which makes
+ the metrics available for metric systems such as Atlassian Statuspage,
+  Prometheus and StatsD.
 
 ## TL;DR
 
 ```console
 helm repo add promitor https://promitor.azurecr.io/helm/v1/repo
-helm install promitor/promitor-agent-scraper
+helm repo update
+helm install promitor-agent-scraper promitor/promitor-agent-scraper
 ```
 
 ## Introduction
@@ -28,7 +29,7 @@ Service so that other Pods can consume it.
 To install the chart with the release name `promitor-agent-scraper`:
 
 ```console
-$ helm install --name promitor-agent-scraper promitor/promitor-agent-scraper \
+$ helm install promitor-agent-scraper promitor/promitor-agent-scraper \
                --set azureAuthentication.appId='<azure-ad-app-id>' \
                --set azureAuthentication.appKey='<azure-ad-app-key>' \
                --values /path/to/metric-declaration.yaml
@@ -42,7 +43,7 @@ declaration, for more information see [our documentation](https://promitor.io/de
 To uninstall/delete the `promitor-agent-scraper` deployment:
 
 ```console
-helm delete promitor-agent-scraper
+helm uninstall promitor-agent-scraper
 ```
 
 The command removes all the Kubernetes components associated with the chart and
@@ -63,13 +64,19 @@ their default values.
 | `resourceDiscovery.enabled`  | Indication whether or not resource discovery is required | `false`            |
 | `resourceDiscovery.host`  | DNS name or IP address of the Promitor Resource Discovery agent |             |
 | `resourceDiscovery.port`  | Port (UDP) address of the Promitor Resource Discovery agent | `80`            |
+| `metricSinks.atlassianStatuspage.enabled` | Indication whether or not metrics should be emitted to a StatsD server | `false`|
+| `metricSinks.atlassianStatuspage.pageId`  | Id of Atlassian Statuspage page |             |
+| `metricSinks.atlassianStatuspage.apiKey`  | API key of Atlassian Statuspage page |             |
+| `metricSinks.atlassianStatuspage.systemMetricMapping`  | Mapping of the Atlassian Statuspage system metrics and Promitor metrics. Learn how to configure it [below](#atlassian-statuspage).| None            |
+| `metricSinks.prometheusScrapingEndpoint.enabled`  | Indication whether or not metrics should be exposed as a Prometheus scraping endpoint | `true`|
+| `metricSinks.prometheusScrapingEndpoint.baseUriPath`  | Path where the scraping endpoint for Prometheus is being exposed | `/metrics`            |
+| `metricSinks.prometheusScrapingEndpoint.enableMetricTimestamps`  | Indication whether or not to include timestamp | `true`            |
+| `metricSinks.prometheusScrapingEndpoint.metricUnavailableValue`  | Value to report in Prometheus when no metric was found whether or not to include timestamp | `NaN`            |
+| `metricSinks.prometheusScrapingEndpoint.enableServiceDiscovery`  | Indication whether or not service discovery with annotations should be enabled ([docs](https://github.com/helm/charts/tree/master/stable/prometheus#scraping-pod-metrics-via-annotations)) | `true`            |
+| `metricSinks.statsd.enabled`  | Indication whether or not metrics should be emitted to a StatsD server | `false`|
 | `metricSinks.statsd.host`  | DNS name or IP address of StatsD server |             |
 | `metricSinks.statsd.port`  | Port (UDP) address of StatsD server | `8125`            |
 | `metricSinks.statsd.metricPrefix`  | Prefix that will be added to every metric defined in the metric declaration |             |
-| `prometheus.scrapeEndpointPath`  | Path where the scraping endpoint for Prometheus is being exposed | `/metrics`            |
-| `prometheus.enableMetricTimestamps`  | Indication whether or not to include timestamp | `true`            |
-| `prometheus.metricUnavailableValue`  | Value to report in Prometheus when no metric was found whether or not to include timestamp | `NaN`            |
-| `prometheus.enableServiceDiscovery`  | Indication whether or not service discovery with annotations should be enabled ([docs](https://github.com/helm/charts/tree/master/stable/prometheus#scraping-pod-metrics-via-annotations)) | `true`            |
 | `telemetry.applicationInsights.enabled`  | Indication whether or not to send telemetry to Azure Application Insights | `false`            |
 | `telemetry.applicationInsights.logLevel`  | Minimum level of logging for Azure Application Insights |             |
 | `telemetry.applicationInsights.key`  | Application Insights instrumentation key |             |
@@ -101,7 +108,7 @@ Specify each parameter using the `--set key=value[,key=value]` argument to
 `helm install`. For example:
 
 ```console
-$ helm install promitor/promitor-agent-scraper --name promitor-agent-scraper \
+$ helm install promitor-agent-scraper promitor/promitor-agent-scraper \
                --set azureAuthentication.appId='<azure-ad-app-id>' \
                --set azureAuthentication.appKey='<azure-ad-app-key>' \
                --set azureMetadata.tenantId='<azure-tenant-id>' \
@@ -113,5 +120,17 @@ Alternatively, a YAML file that specifies the values for the above parameters ca
 be provided while installing the chart. For example,
 
 ```console
-helm install promitor/promitor-agent-scraper --name promitor-agent-scraper -f values.yaml
+helm install promitor-agent-scraper promitor/promitor-agent-scraper -f values.yaml
 ```
+
+### Atlassian Statuspage
+
+You can easily provide system metric mapping between Atlassian Statuspage system metrics and Promitor metrics.
+
+```yaml
+systemMetricMapping: []
+- id: <atlassian-statuspage-system-metric-id>
+  promitorMetricName: <promitor-metric-name>
+```
+
+This defines which Promitor metric should be reported as a given Atlassian Statuspage system metrics.

charts/promitor-agent-scraper/templates/NOTES.txt

@@ -3,14 +3,14 @@
 export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ include "promitor-agent-scraper.name" . }},release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
 kubectl port-forward --namespace {{ .Release.Namespace }} $POD_NAME 8080:{{ .Values.service.targetPort }}
 
-2. Check the scraping output at http://0.0.0.0:8080{{ .Values.prometheus.scrapeEndpointPath }}
+2. Check the scraping output at http://0.0.0.0:8080{{ .Values.metricSinks.prometheusScrapingEndpoint.baseUriPath }}
 
 3. To set up Prometheus in your cluster & pull in metrics from Promitor's scraping output, run:
 
 cat > promitor-scrape-config.yaml <<EOF
 extraScrapeConfigs: |
   - job_name: {{ template "promitor-agent-scraper.fullname" . }}
-    metrics_path: {{ .Values.prometheus.scrapeEndpointPath }}
+    metrics_path: {{ .Values.metricSinks.prometheusScrapingEndpoint.baseUriPath }}
     static_configs:
       - targets:
         - {{ template "promitor-agent-scraper.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.port }}

charts/promitor-agent-scraper/templates/configmap.yaml

@@ -1,4 +1,5 @@
 {{- $metrics := toYaml .Values.metrics -}}
+{{- $systemMetricMapping := toYaml .Values.metricSinks.atlassianStatuspage.systemMetricMapping -}}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -18,7 +19,7 @@ data:
         port: {{ .Values.resourceDiscovery.port }}
   {{- end }}
       metricSinks:
-  {{- if .Values.metricSinks.statsd.host }}
+  {{- if .Values.metricSinks.statsd.enabled }}
         statsd:
           host: {{ .Values.metricSinks.statsd.host | quote }}
   {{- if .Values.metricSinks.statsd.port }}
@@ -28,10 +29,18 @@ data:
           metricPrefix: {{ .Values.metricSinks.statsd.metricPrefix | quote }}
   {{- end }}
   {{- end }}
+  {{- if .Values.metricSinks.prometheusScrapingEndpoint.enabled }}
         prometheusScrapingEndpoint:
-          metricUnavailableValue: {{ .Values.prometheus.metricUnavailableValue | quote }}
-          enableMetricTimestamps: {{ .Values.prometheus.enableMetricTimestamps | quote }}
-          baseUriPath: {{ .Values.prometheus.scrapeEndpointPath | quote }}
+          metricUnavailableValue: {{ .Values.metricSinks.prometheusScrapingEndpoint.metricUnavailableValue | quote }}
+          enableMetricTimestamps: {{ .Values.metricSinks.prometheusScrapingEndpoint.enableMetricTimestamps | quote }}
+          baseUriPath: {{ .Values.metricSinks.prometheusScrapingEndpoint.baseUriPath | quote }}
+  {{- end }}
+  {{- if .Values.metricSinks.atlassianStatuspage.enabled }}
+        atlassianStatuspage:
+          pageId: {{ .Values.metricSinks.atlassianStatuspage.pageId | quote }}
+          systemMetricMapping:
+{{ tpl $systemMetricMapping . | indent 10 }}
+  {{- end }}
       metricsConfiguration:
         absolutePath: /config/metrics-declaration.yaml
       telemetry:

charts/promitor-agent-scraper/templates/deployment.yaml

@@ -53,6 +53,13 @@ spec:
               secretKeyRef:
                   name: {{ template "promitor-agent-scraper.secretname" . }}
                   key: {{ .Values.secrets.appKeySecret }}
+  {{- if .Values.metricSinks.atlassianStatuspage.apiKey }}
+          - name: PROMITOR_ATLASSIAN_STATUSPAGE_APIKEY
+            valueFrom:
+              secretKeyRef:
+                  name: {{ template "promitor-agent-scraper.secretname" . }}
+                  key: {{ .Values.secrets.atlassianStatuspageApiKey }}
+  {{- end }}
           resources:
 {{ toYaml .Values.resources | indent 12 }}
           volumeMounts:

charts/promitor-agent-scraper/templates/secret.yaml

@@ -12,4 +12,7 @@ type: Opaque
 data:
   {{ .Values.secrets.appIdSecret }}: {{ .Values.azureAuthentication.appId | b64enc | quote }}
   {{ .Values.secrets.appKeySecret }}: {{ .Values.azureAuthentication.appKey | b64enc | quote }}
+  {{- if .Values.metricSinks.atlassianStatuspage.apiKey }}
+  {{ .Values.secrets.atlassianStatuspageApiKey }}: {{ .Values.metricSinks.atlassianStatuspage.apiKey | b64enc | quote }}
+  {{- end }}
 {{- end }}

charts/promitor-agent-scraper/templates/service.yaml

@@ -8,10 +8,10 @@ metadata:
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
     type: {{ .Values.service.labelType }}
-{{- if .Values.prometheus.enableServiceDiscovery }}
+{{- if .Values.metricSinks.prometheusScrapingEndpoint.enableServiceDiscovery }}
   annotations:
     prometheus.io/scrape: "true"
-    prometheus.io/path: {{ .Values.prometheus.scrapeEndpointPath | quote }}
+    prometheus.io/path: {{ .Values.metricSinks.prometheusScrapingEndpoint.baseUriPath | quote }}
     prometheus.io/port: {{ .Values.service.targetPort | quote }}
 {{- end }}
 spec:

charts/promitor-agent-scraper/values.yaml

@@ -15,20 +15,28 @@ azureAuthentication:
   appId: ""
   appKey: ""
 metricSinks:
-  useStatsd: false
+  atlassianStatuspage:
+    enabled: false
+    pageId: ""
+    apiKey: ""
+    systemMetricMapping: []
+    # - id: <atlassian-statuspage-system-metric-id>
+    #   promitorMetricName: <promitor-metric-name>
+  prometheusScrapingEndpoint:
+    enabled: true
+    baseUriPath: /metrics
+    enableMetricTimestamps: true
+    metricUnavailableValue: NaN
+    enableServiceDiscovery: true
   statsd:
+    enabled: false
     host: ""
     port: 8125
     metricPrefix: ""
 resourceDiscovery:
   enabled: false
   host: ""
   port: 80
-prometheus:
-  scrapeEndpointPath: /metrics
-  enableMetricTimestamps: true
-  metricUnavailableValue: NaN
-  enableServiceDiscovery: true
 telemetry:
   defaultLogLevel: "Error"
   applicationInsights:
@@ -70,6 +78,7 @@ secrets:
   appIdSecret: azure-app-id
   appKeySecret: azure-app-key
   appInsightsSecret: azure-appinsights-key
+  atlassianStatuspageApiKey: atlassian-statuspage-apikey
 
 service:
   exposeExternally: False

docs/deployment/scraper/index.md

@@ -31,6 +31,12 @@ Install the Promitor Chart repository:
 ❯ helm repo add promitor https://promitor.azurecr.io/helm/v1/repo
 ```
 
+Refresh your local Chart repositories:
+
+```shell
+❯ helm repo update
+```
+
 If all goes well you should be able to list all Promitor charts:
 
 ```shell
@@ -54,13 +60,21 @@ azureMetadata:
 
 runtime:
   metricSinks:
+    atlassianStatuspage:
+      enabled: true
+      pageId: "ABC"
+      systemMetricMapping:
+      - id: nfkgnrwpn545
+        promitorMetricName: promitor_demo_appplan_percentage_cpu
+    prometheusScrapingEndpoint:
+      enabled: true
+      baseUriPath: /metrics
+      enableMetricTimestamps: True
     statsd:
+      enabled: true
       host: graphite
       port: 8125
       metricPrefix: poc.promitor.
-  prometheus:
-    scrapeEndpointPath: /metrics
-    enableMetricTimestamps: True
   telemetry:
     applicationInsights:
       enabled: True
@@ -85,7 +99,7 @@ If you have a `metric-declaration.yaml` file, you can create a basic deployment
 with this command:
 
 ```shell
-❯ helm install --name promitor-agent-scraper promitor/promitor-agent-scraper \
+❯ helm install promitor-agent-scraper promitor/promitor-agent-scraper \
                --set azureAuthentication.appId='<azure-ad-app-id>' \
                --set azureAuthentication.appKey='<azure-ad-app-key>' \
                --values /path/to/helm-configuration.yaml

docs/walkthrough/scrape-promitor-with-prometheus-on-azure-kubernetes-service.md

@@ -28,7 +28,6 @@ we're monitoring.
   - [Create an AKS Cluster](#create-an-aks-cluster)
 - **[Cluster Setup](#cluster-setup)**
   - [Get credentials](#get-credentials)
-  - [Set up Helm and Tiller](#set-up-helm-and-tiller)
 - **[Deploy Promitor and Prometheus](#deploy-promitor-and-prometheus)**
   - [Create a metrics declaration for Promitor](#create-a-metrics-declaration-for-promitor)
   - [Deploy Promitor to your cluster using Helm](#deploy-promitor-to-your-cluster-using-helm)
@@ -155,48 +154,6 @@ as your current context for all `kubectl` commands.
 Verify your credentials and check that your cluster is up and running with
 `kubectl get nodes`.
 
-### Set up Helm and Tiller
-
-You'll use Helm to install Tiller, the server-side component of Helm. For clusters
-with RBAC (Role-Based Access Control, which is enabled by default on AKS clusters),
-you'll need to set up a service account for Tiller.
-
-Create a file called `helm-rbac.yaml` with the following:
-
-```YAML
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: tiller
-  namespace: kube-system
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: tiller
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: cluster-admin
-subjects:
-  - kind: ServiceAccount
-    name: tiller
-    namespace: kube-system
-```
-
-Create the service account and role binding specified in the above file with the
-`kubectl apply` command:
-
-```bash
-kubectl apply -f helm-rbac.yaml
-```
-
-Then deploy Tiller into your AKS cluster using the `helm init` command:
-
-```bash
-helm init --service-account tiller
-```
-
 ## Deploy Promitor and Prometheus
 
 ### Create a metrics declaration for Promitor
@@ -238,13 +195,13 @@ To deploy, we'll first add the Promitor chart repository to helm:
 
 ```bash
 helm repo add promitor https://promitor.azurecr.io/helm/v1/repo
+helm repo update
 ```
 
 With this repository added, we can deploy Promitor:
 
 ```bash
-helm install promitor/promitor-agent-scraper \
-  --name promitor-agent-scraper \
+helm install promitor-agent-scraper promitor/promitor-agent-scraper \
   --values your/path/to/metric-declaration.yaml
 ```
 
@@ -342,7 +299,7 @@ Grafana's chart has a few default values you may not want long term - persistant
 storage is disabled and admin username/password is randomly generated - but for
 our sample the out-of-the-box install will work.
 
-Run `helm install stable/grafana --name grafana` and you should see output that
+Run `helm install grafana stable/grafana` and you should see output that
 includes this command:
 
 ```shell