This repository has been archived by the owner on May 25, 2023. It is now read-only.
Add performance metrics for scheduling #592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
do-not-merge/work-in-progress
k8s-ci-robot
added
the
size/XXL
|
|
/cc @k82cn @jiaxuanzhou |
|
@Jeffwan: GitHub didn't allow me to request PR reviews from the following users: jiaxuanzhou. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Jeffwan
commented
Feb 14, 2019
| @@ -157,6 +161,7 @@ func (alloc *allocateAction) Execute(ssn *framework.Session) { | |||
| } | |||
|
|
|||
| assigned = true | |||
| metrics.UpdateTaskScheduleDuration(metrics.Duration(taskScheduleStart)) | |||
There was a problem hiding this comment.
Is this the right way to measure kube_batch_task_scheduling_latency_microseconds ?
There was a problem hiding this comment.
no, it is not the right way to measure the latency, the start time should be measured at the point when the task(pod) observed by kube-batch.
There was a problem hiding this comment.
and it is the same way of measuring the latency of job
There was a problem hiding this comment.
the start time should be measured at the point when the task(pod) observed by kube-batch
+1
Maybe we can add more info into TaskInfo.
There was a problem hiding this comment.
If we consider all staged, I think we can add CreationTimestamp in TaskInfo, when kube-batch create task, deep copy timestamp to task. This timestamp would be the startTime.
We probably want to add endTime inside job_info.go#UpdateTaskStatus. If status is Binding, we measure latency. Any other status indicate success of scheduling? I am not sure what's Pipeline. Should we also consider it?
There was a problem hiding this comment.
I think we can add
CreationTimestampin TaskInfo ...
If so, how to handle scheduler crash case? Maybe we can use Pod's CreationTimestamp directly; it only include the additional time from apiserver to scheduler which should be short.
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Jeffwan, k82cn The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
Sorry for leaving this hanging for so long. I will clean it up and submit revision today. |
Jeffwan
changed the title
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
|
Found a problem if pod running on different machines, format time using creationTime: 2019-03-09T19:38:56-08:00 I did some search and notice you already report this problem here https://groups.google.com/forum/#!topic/kubernetes-dev/rUniUqhI5YM Seems it's user's responsibility to keep clock sync. |
Yes, it dependent on user's environment as the timestamp maybe set by different components. |
|
/lgtm |
k8s-ci-robot
added a commit
that referenced
this pull request
Mar 10, 2019
…ream-release-0.4 Automated cherry pick of #592: Update prometheus vendor libs
kevin-wangzefeng
pushed a commit
to kevin-wangzefeng/scheduler
that referenced
this pull request
Jun 28, 2019
Add performance metrics for scheduling
kevin-wangzefeng
pushed a commit
to kevin-wangzefeng/scheduler
that referenced
this pull request
Jun 28, 2019
Add performance metrics for scheduling
kevin-wangzefeng
pushed a commit
to kevin-wangzefeng/scheduler
that referenced
this pull request
Jun 28, 2019
Add performance metrics for scheduling
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Add metrics support for scheduling and use instrumentation to help debug performance issues.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Follow up on #579 Fixes #487
Special notes for your reviewer:
For action, plugin duration, use microseconds,
For e2e, use milliseconds,
Use prometheus.ExponentialBuckets(5, 2, 10) for histogram buckets.
Please check attached metrics make sense to you.
Release note: