Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring guide is broken #2192

Closed
1 task done
kingdonb opened this issue Dec 8, 2021 · 6 comments · Fixed by #2195
Closed
1 task done

Monitoring guide is broken #2192

kingdonb opened this issue Dec 8, 2021 · 6 comments · Fixed by #2195

Comments

@kingdonb
Copy link
Member

kingdonb commented Dec 8, 2021

Describe the bug

We're just doing an internal demo and we noticed that the Monitoring guide appears to be broken. The Flux Control Plane dashboard appears on the Grafana deployment as expected, but none of the pods match.

I have a customized version of the manifests from a fork, which could have introduced problems, but we had someone follow along with the instructions from https://fluxcd.io/docs/guides/monitoring/ and they had similar results. Metrics are not being collected from the Flux pods for some reason.

@Nalum found the issue that I was experiencing was not unique to my customized version. I have one custom pod and that pod was scraped successfully for some reason. His installation had zero customized pods and zero pods were scraped.

Steps to reproduce

Follow the monitoring guide. Notice that metrics are not being collected for the flux-system pods as expected.

Expected behavior

Expected to see that Flux control plane was being monitored and metrics are collected. As a double-check, I deleted all the pods in the cluster and allowed them to restart. It didn't change anything.

Screenshots and recordings

Screen Shot 2021-12-08 at 12 41 31 PM

OS / Distro

MacOS 12.0.1 (Monterey)

Flux version

v0.24.0

Flux check

$ flux check
► checking prerequisites
✔ Kubernetes 1.21.7 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.14.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.18.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.14.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.18.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.19.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.19.0
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@stefanprodan
Copy link
Member

Duplicate of #2150

@stefanprodan stefanprodan marked this as a duplicate of #2150 Dec 8, 2021
@stefanprodan
Copy link
Member

To fix this we need to add each controller with its own key in the pod monitor.

@Nalum
Copy link
Member

Nalum commented Dec 8, 2021

I can look at picking this up if it's no one else has it on their plate

@stefanprodan
Copy link
Member

As for the Kubernetes API req duration, that's gone, we have to delete that graph see #1479

@stefanprodan
Copy link
Member

I can look at picking this up if it's no one else has it on their plate

@Nalum that would be great, thank you!

@stefanprodan
Copy link
Member

Ok now that scraping is sorted, we need to delete the Kubernetes API req duration graph to close this issue.

grafjo pushed a commit to grafjo/flux2 that referenced this issue Dec 12, 2021
this release contains the prometheus operator in version 0.52.1

see fluxcd#2192
fluxcd#2193 for issues
grafjo pushed a commit to grafjo/flux2 that referenced this issue Dec 12, 2021
this release contains the prometheus operator in version 0.52.1

see fluxcd#2192
fluxcd#2193 for issues

Signed-off-by: Johannes Graf <graf@synyx.de>
souleb pushed a commit to souleb/flux2 that referenced this issue Jul 10, 2023
this release contains the prometheus operator in version 0.52.1

see fluxcd#2192
fluxcd#2193 for issues

Signed-off-by: Johannes Graf <graf@synyx.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants