Release v2.0.0

Monitoring Core Module Release 2.0.0

Welcome to the latest release of the monitoring core module of the Kubernetes Fury Distribution, maintained by team SIGHUP.

This release includes upgrades for the components in the module to its latest stable releases, introduces two new components and deprecates other two, it also introduces support for Kubernetes v1.24.0. See more details in the Features section.

Component Images 🚢

Component	Supported Version	Previous Version
alertmanager	v0.24.0	v0.23.0
blackbox-exporter	v0.21.0	New Package
grafana	v8.5.5	v8.3.3
goldpinger	Removed	v3.3.0
kube-rbac-proxy	v0.12.0	v0.11.0
kube-state-metrics	v2.5.0	v2.3.0
node-exporter	v1.3.1	No Update
metrics-server	Removed	0.5.2
prometheus-adapter	v0.9.1	New Package
prometheus-operator	v0.57.0	v0.53.1
prometheus	v2.36.1	v2.32.1
thanos	v0.24.0	No Update
x509-exporter	v3.2.0	2.12.1

Please refer to the individual release notes to get more information on each release.

Breaking Changes 💔

• This is a new major release that involves the deletion and recreation of some resources. More details are in the Update Guide section.
• Since the Prometheus CRD is too big, you will need to use kubectl apply --server-side instead of plain kubectl apply to deploy the module. See prometheus-operator/prometheus-operator#4439 and kubernetes/kubernetes#82292.
• metrics-server is not supported anymore by this module.
• goldpinger is not supported anymore by this module.

Features 💥

• Sync module with release 0.11.0 of the upstream project kube-prometheus.
• Replace metrics-server with prometheus-adapter thus removing the dependency on the cert-manager package and enabling workload autoscaling based on custom metrics.
• Removed goldpinger that was creating high-cardinality metrics in big clusters.
• Added a default configuration con Alertmanager through the usage of the new AlertmanagerConfig CRD.
• Added blackbox-exporter to allow blackbox probing via the Probe CRD.
• Added support for Kubernetes v1.24.x

Update Guide 🦮

Warnings

• Since the release ships changes to some immutable fields, the upgrade process will involve the deletion and recreation of some resources.
• Notice that all the kubectl apply commands make use of the --server-side flag, this is needed because some objects are very large and the annotations introduced by the client side apply go beyond the maximum size.
• AlertManager's configuration has been changed. You'll need to create 3 new secrets with the notification endpoints for it to work properly.

Process

To upgrade this core module from v1.14.x to v2.0.0, execute the following:

• prometheus-operator

kubectl delete deployments.apps prometheus-operator -n monitoring
kustomize build katalog/promethes-operator | kubectl apply -f - --server-side

• prometheus-operated

kubectl delete poddisruptionbudgets.policy prometheus-k8s -n monitoring
kubectl delete clusterrolebinding.rbac.authorization.k8s.io prometheus-k8s-scrape
kubectl delete clusterroles.rbac.authorization.k8s.io prometheus-k8s-scrape
kubectl delete prometheusrules.monitoring.coreos.com prometheus-k8s-rules -n monitoring
kustomize build katalog/prometheus-operated | kubectl apply -f - --server-side

• alertmanager-operated

Alert Manager now reads the notification endpoint for Slack and Healthchecks.io from secrets, instead of having it embedded in the configuration.

You will need to create these secrets before applying the new version:

💡 the next commands are provided as examples, we recommend keeping the generation of the secrets together with the rest of your Infrastructure as Code files.

$ kubectl create secret generic infra-slack-webhook -n monitoring --from-literal url="<http(s)://your endpoint URL>"
secret/infra-slack-webhook created

$ kubectl create secret generic healthchecks-webhook -n monitoring --from-literal url="<http(s)://your endpoint URL>"
secret/healthchecks-webhook created

$ kubectl create secret generic k8s-slack-webhook -n monitoring --from-literal url="<http(s)://your endpoint URL>"
secret/k8s-slack-webhook created

Proceed with the upgrade:

kubectl delete poddisruptionbudget.policy alertmanager-main -n monitoring
kustomize build katalog/alertmanager-operated | kubectl apply -f - --server-side

• blackbox-exporter

kustomize build katalog/blackbox-exporter | kubectl apply -f - --server-side

• goldpinger

kubectl delete servicemonitor.monitoring.coreos.com goldpinger -n monitoring
kubectl delete service goldpinger -n monitoring
kubectl delete daemonset.apps goldpinger -n monitoring
kubectl delete clusterrole.rbac.authorization.k8s.io goldpinger
kubectl delete serviceaccount goldpinger -n monitoring
kubectl delete rolebinding.rbac.authorization.k8s.io goldpinger:cluster:view -n monitoring

• grafana

kubectl delete deployments.apps grafana -n monitoring
kustomize build katalog/grafana | kubectl apply -f - --server-side

• kube-proxy-metrics

kustomize build katalog/kube-proxy-metrics | kubectl apply -f - --server-side --force-conflicts

• kube-state-metrics

kubectl delete deployments.apps kube-state-metrics -n monitoring
kustomize build katalog/kube-state-metrics | kubectl apply -f - --server-side

• metrics-server

kubectl delete apiservice.apiregistration.k8s.io v1beta1.metrics.k8s.io
kubectl delete service metrics-server -n kube-system
kubectl delete deployment.apps metrics-server -n kube-system
kubectl delete clusterrolebinding.rbac.authorization.k8s.io metrics-server:system:auth-delegator
kubectl delete clusterrolebinding.rbac.authorization.k8s.io system:metrics-server
kubectl delete clusterrole.rbac.authorization.k8s.io system:aggregated-metrics-reader
kubectl delete clusterrole.rbac.authorization.k8s.io system:metrics-server
kubectl delete rolebinding.rbac.authorization.k8s.io metrics-server-auth-reader -n kube-system
kubectl delete serviceaccount metrics-server -n kube-system
kubectl delete certificate.cert-manager.io metrics-server-tls -n kube-system
kubectl delete certificate.cert-manager.io metrics-server-ca -n kube-system
kubectl delete issuer.cert-manager.io metrics-server-ca -n kube-system
kubectl delete issuer.cert-manager.io metrics-server-selfsign -n kube-system
kubectl delete secret metrics-server-ca metrics-server-tls -n kube-system

• node-exporter

kubectl delete daemonsets.apps node-exporter -n monitoring
kustomize build katalog/node-exporter | kubectl apply -f - --server-side

• prometheus-adapter

kustomize build katalog/prometheus-adapter | kubectl apply -f - --server-side

• x509-exporter

kubectl delete serviceaccount x509-certificate-exporter-node -n monitoring
kubectl delete clusterrole.rbac.authorization.k8s.io x509-certificate-exporter-node
kubectl delete clusterrolebinding.rbac.authorization.k8s.io x509-certificate-exporter-node
kubectl delete daemonset.apps x509-certificate-exporter-nodes -n monitoring
kustomize build katalog/x509-exporter | kubectl apply -f - --server-side

Release v1.14.3

Monitoring Core Module version v1.14.3

This patch includes hotfixes in alertmanager-operated alert template and alerts annotations, namely the renaming from annotations.message to annotations.description.

Component Images 🚢

Component	Supported Version	Previous Version
prometheus-operator	v0.53.1	No update
prometheus	v2.32.1	No update
alertmanager	v0.23.0	No update
grafana	v8.3.3	No update
goldpinger	v3.3.0	No update
kube-rbac-proxy	v0.11.0	No update
kube-state-metrics	v2.3.0	No update
node-exporter	v1.3.1	No update
metrics-server	0.5.2	No update
thanos	v0.24.0	No update
x509-exporter	v2.12.1	No update

Please refer to the individual release notes to get more detailed information on each release.

Bug Fixes

• #101 Fix an issue with the default Alertmanager template and with alerts annotations.

Update Guide 🦮

Process

To upgrade this core module from v1.14.2 to v1.14.3, you need to download this new version, and then apply the kustomize project. No further action is required.

kustomize build katalog/alertmanager-operated | kubectl apply -f -
kustomize build katalog/prometheus-operated | kubectl apply -f -
kustomize build katalog/kubeadm-sm | kubectl apply -f - # if needed

Release v1.13.1

Monitoring Core Module version v1.13.1

This patch includes hotfixes in alertmanager-operated alert template and alerts annotations, namely the renaming from annotations.message to annotations.description.

Upgrade path

To upgrade this core module from v1.13.0 to v1.13.1:

1. Download the new module version
2. Build and apply the kustomize project:

kustomize build katalog/alertmanager-operated | kubectl apply -f -
kustomize build katalog/prometheus-operated | kubectl apply -f -
kustomize build katalog/kubeadm-sm | kubectl apply -f - # if needed

Release v1.14.2

Monitoring Core Module version v1.14.2

This patch includes hotfix in x509-exporter package.

Component Images 🚢

Component	Supported Version	Previous Version
prometheus-operator	v0.53.1	No update
prometheus	v2.32.1	No update
alertmanager	v0.23.0	No update
grafana	v8.3.3	No update
goldpinger	v3.3.0	No update
kube-rbac-proxy	v0.11.0	No update
kube-state-metrics	v2.3.0	No update
node-exporter	v1.3.1	No update
metrics-server	0.5.2	No update
thanos	v0.24.0	No update
x509-exporter	v2.12.1	No update

Please refer the individual release notes to get a detailed info on the
releases.

Bug Fixes

• #88 Fix an issue
  on x509-exporter changing the --port=9793 flag to
  --listen-address=:9793.

Update Guide 🦮

Process

To upgrade this core module from v1.14.1 to v1.14.2, you need to download this new version, then apply the
kustomize project. No further action is required.

kustomize build katalog/x509-exporter | kubectl apply -f -

Release v1.14.1

Monitoring Core Module version v1.14.1

This is a patch release reverts the commonLabels applied in v1.14.0 because they break updating the module in the future.

Update Guide 🦮

Warnings

• Since the release rollbacks some changes to immutable fields, if deployments, statefulset and daemonsets, are not deleted first before applying the module, it will error out. Check the Process below for more info.

Process

If you are upgrading from version v1.14.0 to v1.14.1 you need to download this new version and then apply the kustomize project as shown below.

There will be downtime on the components.

# Delete the deployments with the labels so they can be upgraded
kubectl delete deployment.apps/prometheus-operator -n monitoring
# Finally, apply the new version
kustomize build katalog/prometheus-operator | kubectl apply -f -
# Repeat for each package
kubectl delete statefulset.apps/prometheus-k8s -n monitoring
kustomize build katalog/prometheus-operated | kubectl apply -f -
kubectl delete deployment.apps/grafana -n monitoring
kustomize build katalog/grafana | kubectl apply -f -
kubectl delete -n kube-system deployment.apps/metrics-server
kustomize build katalog/metrics-server | kubectl apply -f -
kubectl delete daemonset.apps/goldpinger -n monitoring
kustomize build katalog/goldpinger | kubectl apply -f-
kustomize build katalog/kubeadm-sm | kubectl apply -f-
kubectl delete daemonset.apps/kube-proxy-metrics -n monitoring
kustomize build katalog/kube-proxy-metrics | kubectl apply -f-
kubectl delete deployment.apps/kube-state-metrics -n monitoring
kustomize build katalog/kube-state-metrics | kubectl apply -f-
kubectl delete daemonset.apps/node-exporter -n monitoring
kustomize build katalog/node-exporter | kubectl apply -f-
kubectl delete statefulset.apps/alertmanager-main -n monitoring
kustomize build katalog/alertmanager-operated | kubectl apply -f-

If you are upgrading from a version < v1.14.0, you can simply apply the kustomize project as shown below.

kustomize build katalog/prometheus-operator | kubectl apply -f -
kustomize build katalog/prometheus-operated | kubectl apply -f -
kustomize build katalog/node-exporter | kubectl apply -f -
kustomize build katalog/metrics-server | kubectl apply -f -
kustomize build katalog/kubeadm-sm | kubectl apply -f -
kustomize build katalog/kube-state-metrics | kubectl apply -f -
kustomize build katalog/kube-proxy-metrics | kubectl apply -f -
kustomize build katalog/grafana | kubectl apply -f -
kustomize build katalog/goldpinger | kubectl apply -f -
kustomize build katalog/gke-sm | kubectl apply -f -
kustomize build katalog/eks-sm | kubectl apply -f -
kustomize build katalog/alertmanager-operated | kubectl apply -f -
kustomize build katalog/aks-sm | kubectl apply -f -

Release v1.14.0

❌ This release contains issues, please use the version v1.14.1 instead ❌

# Monitoring Core Module Release 1.14.0

Welcome to the latest release of monitoring module of Kubernetes Fury Distribution maintained by team
SIGHUP.

This latest release is an attempt on upgrading the components in the module to
its latest stable release along with adding the tech preview of the latest
kubernetes release v1.23.0.

Component Images 🚢

Component	Supported Version	Previous Version
prometheus-operator	v0.53.1	v0.50.0
prometheus	v2.32.1	v2.29.1
alertmanager	v0.23.0	v0.22.2
grafana	v8.3.3	v8.1.2
goldpinger	v3.3.0	v3.2.0
kube-rbac-proxy	v0.11.0	No update
kube-state-metrics	v2.3.0	v2.2.0
node-exporter	v1.3.1	1.2.2
metrics-server	0.5.2	0.5.0
thanos	v0.24.0	v0.22.0
x509-exporter	v2.12.1	2.11.0

Please refer the individual release notes to get a detailed info on the
releases.

Known Issues ⚠️

• This release involves recreation of certain resources like daemonset, deployments, etc. We recommend deleting it with --cascade=orphan, to avoid changes being made to the pods.
• We drop support of Kubernetes versions <= v1.19.x

Breaking Changes 💔

• #77 Added Kubernetes labels to all the components of the module: Since labels are immutable fields in deployments, daemonsets, etc., this change requires a recreation of those resources.
• component breaking changes:
  • thanos: breaking ⚠️ Add --endpoint flag to the querier. The --store flag will eventually be replaced .

Features 💥

• #78 Sync module with the latest kube-prometheus updates
• #77 Added Makefile, JSON builder and .bumpversion config to the module
• #76 Upgrade the component images
• #75 Added e2e-test support for k8s runtime 1.23
• #80 Updating the README structure for monitoring module

Update Guide 🦮

Warnings

• Since the release ships changes to some immutable fields, if deployments and daemonsets, are not deleted first before applying the module, it will error out, we advice deletion of those objects with the flag cascade=orphan

Process

To upgrade this core module from v1.13.x to v1.14.0, you need to download this new version, then apply the kustomize project.

kubectl delete deployment.apps/prometheus-operator -n monitoring --cascade=orphan
kustomize build katalog/prometheus-operator | kubectl apply -f -
kubectl delete statefulset.apps/prometheus-k8s -n monitoring --cascade=orphan
kustomize build katalog/prometheus-operated | kubectl apply -f -
kubectl delete deployment.apps/grafana -n monitoring --cascade=orphan
kustomize build katalog/grafana | kubectl apply -f -
kubectl delete -n kube-system deployment.apps/metrics-server --cascade=orphan
kustomize build katalog/metrics-server | kubectl apply -f -
kubectl delete daemonset.apps/goldpinger -n monitoring --cascade=orphan
kustomize build katalog/goldpinger | kubectl apply -f-
kustomize build katalog/kubeadm-sm | kubectl apply -f-
kubectl delete deployment.apps/kube-proxy-metrics -n monitoring --cascade=orphan
kustomize build katalog/kube-proxy-metrics | kubectl apply -f-
kubectl delete daemonset.apps/kube-state-metrics -n monitoring --cascade=orphan
kustomize build katalog/kube-state-metrics | kubectl apply -f-
kubectl delete daemonset.apps/node-exporter -n monitoring --cascade=orphan
kustomize build katalog/node-exporter | kubectl apply -f-
kubectl delete statefulset.apps/alertmanager-main -n monitoring --cascade=orphan
kustomize build katalog/alertmanager-operated | kubectl apply -f-

Release v1.13.0

Monitoring Core Module version 1.13.0

fury-kubernetes-monitoring is part of the SIGHUP maintained Kubernetes Fury Distribution. The module ships all necessary components to deploy monitoring tools on top of Kubernetes primarily based off of prometheus. Team SIGHUP makes it a priority to maintain these modules in compliance with CNCF and with all the latest features from upstream.

This release introduces the support for Kubernetes runtime 1.22 and drops support for 1.18. Refer the Compatibility Matrix for more.

Changelog

Breaking changes

None

Features

• #71 Upgrading the component images:
  • Prometheus Operator from 0.48.1 to 0.50.0
  • Prometheus from 2.22.2 to 2.29.1
  • Grafana from 7.5.7 to 8.1.2
  • kube-proxy-metrics 0.10.0 to 0.11.0
  • node-exporter 1.1.2 to 1.2.2
  • Thanos 0.20.2 to 0.22.0
  • x509-exporter 2.9.2 to 2.11.0
• #70 Adding support for Kubernetes 1.22 in e2e tests
• #71 Remove CPUThrottlingHigh as it was previously removed
• #71 Prometheus ruleSelector defaults to all rules
• #71 Updating the CRDs to v1 version and minor fixes to alerts
• #72 Syncing the dashboards with kube-prometheus
• #72 Adding podDisruptionBudget for alertmanager and prometheus
• #72 Adding alertmanager dashboard

Bug fixes

None

Security Fixes

None

Documentation updates

• #65 Add a note for PING_NUMBER env var of Goldpinger. Thanks to @beratio

Known Issues

When applying the first time, prometheus-operator and prometheus-operated has circular dependency. So make sure to apply both of them twice.

Upgrade Guide

Warnings

• It is noted that the Networking dashboards tend to take a bit longer upon the first setup to update data.
• The upgrade process should be faily smooth.

Process

To upgrade this core module from v1.12.X to v1.13.0, you need to download this new version, then apply the
kustomize project. No further action is required.

kustomize build katalog/prometheus-operator | kubectl apply -f -
kustomize build katalog/prometheus-operated | kubectl apply -f -
kustomize build katalog/node-exporter | kubectl apply -f -
kustomize build katalog/metrics-server | kubectl apply -f -
kustomize build katalog/kubeadm-sm | kubectl apply -f -
kustomize build katalog/kube-state-metrics | kubectl apply -f -
kustomize build katalog/kube-proxy-metrics | kubectl apply -f -
kustomize build katalog/grafana | kubectl apply -f -
kustomize build katalog/goldpinger | kubectl apply -f -
kustomize build katalog/gke-sm | kubectl apply -f -
kustomize build katalog/eks-sm | kubectl apply -f -
kustomize build katalog/alertmanager-operated | kubectl apply -f -
kustomize build katalog/aks-sm | kubectl apply -f -

Release v1.12.3

Monitoring Core Module version v1.12.3

This patch includes hotfix in thanos/thanos-with-store package.

Changelog

• Remove unsupported flag --experimental.enable-index-cache-postings-compression (behaviour is now the default)

Upgrade path

To upgrade this core module from v1.12.2 to v1.12.3:

1. Download the new module version
2. Build and apply the kustomize project:

kustomize build katalog/thanos/thanos-with-store | kubectl apply -f -

Release v1.12.2

Monitoring Core Module version v1.12.2

This patch includes hotfix in kube-rbac-proxy containers.

Changelog

• Fixing the UID:GID in kube-rbac-proxy
  • brancz/kube-rbac-proxy/Dockerfile
  • Was configured 65534:65534, correct one is: 65532:65532

Upgrade path

To upgrade this core module from v1.12.1 to v1.12.2, you need to download this new version, then apply the
kustomize project. No further action is required.

kustomize build katalog/kube-proxy-metrics | kubectl apply -f -
kustomize build katalog/node-exporter | kubectl apply -f -

Release v1.12.1

Monitoring Core Module version v1.12.1

This patch includes a couple of fixes in prometheus and the kube-proxy-metrics.

Changelog

• Fixing the GID in kube-rbac-proxy #66. Thanks to @nandajavarma
• changed severity when all targets are down #65. Thanks to @lzecca78

Upgrade path

To upgrade this core module from v1.12.0 to v1.12.1, you need to download this new version, then apply the
kustomize project. No further action is required.

kustomize build katalog/kube-proxy-metrics | kubectl apply -f -
kustomize build katalog/prometheus-operated | kubectl apply -f -