Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Commit

Permalink
metrics: Add condition gauge
Browse files Browse the repository at this point in the history
Fixes #395
  • Loading branch information
seaneagan committed May 5, 2020
1 parent 72a3dc9 commit aa13b68
Show file tree
Hide file tree
Showing 3 changed files with 64 additions and 16 deletions.
33 changes: 17 additions & 16 deletions docs/references/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,27 +11,28 @@ in Prometheus format.
| `release_count` | Count of releases managed by the operator. |
| `release_duration_seconds` | Release synchronization duration in seconds. This duration includes one or many `release_phase_durations`. |
| `release_phase_duration_seconds` | Release phase synchronization duration in seconds. |
| `release_phase_info` | The (negative) integer equaling the current phase of a release. Negative values are failed phases, `0` equals to unknown. See [release phases](#release-phases).
| `release_condition_info` | Release condition status gauge, see [release conditions](#release-conditions).
| `release_queue_length_count` | Count of release jobs waiting in the queue to be processed. |

### Release conditions

### Release phases
#### Labels

The following is a table of the values the `release_phase_info` metric exposes,
and the phase they represent:
| Label | Label Value |
|----------------|---
| `namespace` | `metadata.namespace` of `HelmRelease`
| `name` | `metadata.name` of `HelmRelease`
| `condition` | [condition type](helmrelease-custom-resource.md#helm.fluxcd.io/v1.HelmReleaseConditionType)

| Value | Phase |
#### Values

Values represent the [condition status](helmrelease-custom-resource.md#helm.fluxcd.io/v1.ConditionStatus).

| Value | Condition Status |
|-------|---
| `-4` | `ChartFetchFailed`
| `-3` | `Failed`
| `-2` | `RollbackFailed`
| `-1 ` | `RolledBack`
| `-1` | `False`
| `0` | `Unknown`
| `1` | `RollingBack`
| `2` | `Installing`
| `3` | `Upgrading`
| `4` | `ChartFetched`
| `5` | `Succeeded`
| `1` | `True`

## Prometheus alert rules examples

Expand All @@ -51,12 +52,12 @@ for: 30m

```yaml
alert: HelmReleaseRolledBack
expr: flux_helm_operator_release_phase_info == -1
expr: flux_helm_operator_release_condition_info{condition="RolledBack"} == 1
```

### `HelmRelease` subject to an error

```yaml
alert: HelmReleaseError
expr: flux_helm_operator_release_phase_info < -1
expr: flux_helm_operator_release_phase_info{condition="Released"} == -1
```
1 change: 1 addition & 0 deletions pkg/status/conditions.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ func SetCondition(client v1client.HelmReleaseInterface, hr *v1.HelmRelease, cond
set(cHr)
}

ObserveReleaseConditions(*hr, *cHr)
_, err = client.UpdateStatus(cHr)
firstTry = false
return
Expand Down
46 changes: 46 additions & 0 deletions pkg/status/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package status

import (
v1 "github.com/fluxcd/helm-operator/pkg/apis/helm.fluxcd.io/v1"
"github.com/go-kit/kit/metrics/prometheus"
stdprometheus "github.com/prometheus/client_golang/prometheus"
)

const (
LabelNamespace = "namespace"
LabelName = "name"
LabelCondition = "condition"
)

var (
conditionStatusToGaugeValue = map[v1.ConditionStatus]float64{
v1.ConditionFalse: -1,
v1.ConditionUnknown: 0,
v1.ConditionTrue: 1,
}
releaseCondition = prometheus.NewGaugeFrom(stdprometheus.GaugeOpts{
Namespace: "flux",
Subsystem: "helm_operator",
Name: "release_condition_info",
Help: "Current HelmRelease condition status. Values are -1 (false), 0 (unknown or absent), 1 (true)",
}, []string{LabelNamespace, LabelName, LabelCondition})
)

func ObserveReleaseConditions(old v1.HelmRelease, new v1.HelmRelease) {
conditions := make(map[v1.HelmReleaseConditionType]v1.ConditionStatus)
for _, condition := range old.Status.Conditions {
// Initialize conditions from old status to unknown, so that if
// they are removed in new status, they do not contain stale data.
conditions[condition.Type] = v1.ConditionUnknown
}
for _, condition := range new.Status.Conditions {
conditions[condition.Type] = condition.Status
}
for conditionType, conditionStatus := range conditions {
releaseCondition.With(
LabelNamespace, new.Namespace,
LabelName, new.Name,
LabelCondition, string(conditionType),
).Set(conditionStatusToGaugeValue[conditionStatus])
}
}

0 comments on commit aa13b68

Please sign in to comment.