-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ Add metrics for total workers and active workers #1125
✨ Add metrics for total workers and active workers #1125
Conversation
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/assign @estroz |
33ce5b6
to
5cefa42
Compare
/retest |
1 similar comment
/retest |
@@ -169,6 +169,7 @@ func (c *Controller) Start(stop <-chan struct{}) error { | |||
|
|||
// Launch workers to process resources | |||
c.Log.Info("Starting workers", "worker count", c.MaxConcurrentReconciles) | |||
ctrlmetrics.WorkerCount.WithLabelValues(c.Name).Set(float64(c.MaxConcurrentReconciles)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you explain the logic behind collecting this metric?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- It allows correlating other metrics to the number of workers, for example workqueue_depth
- It allows calculating the
available_workers
@estroz PTAL |
Without knowing too much about how metrics are collected this looks fine to me. However there are now 7 metrics collected for a controller. Will an increase in the number of metrics impact cluster resources significantly? /lgtm |
Cancel whenever /hold |
No, the way this works is that only the current value of a metric is stored and Prometheus is expected to collect it. The thing to keep in mind is to not expose too many metrics which usually happens when using labels that have a yet-to-be-determined set of values, because every metric+label combinaation actually ends up being an unique metric in Prometheus: https://www.robustperception.io/cardinality-is-key /hold cancel |
/retest |
1 similar comment
/retest |
Helps debugging issues around reconciliation duration. Right now its impossible to find out the number of workers from metrics and its also impossible to find out the number of active workers. The only thing we have is
workqueue_unfinished_work_seconds
which if big is probably not good but its not clearly indicating if we have one worker that takes longer, if all workers are blocked etc.