RHOAIENG-4198: Initial commit for kserve perf metrics #396
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
||
| You can monitor the following metrics for a specific model that is deployed on the single-model serving platform: | ||
|
|
||
| * *Number of requests* - The number of HTTP requests that have failed or succeeded for a specific model. |
There was a problem hiding this comment.
Suggested change
| * *Number of requests* - The number of HTTP requests that have failed or succeeded for a specific model. | |
| * *Number of requests* - The number of requests that have failed or succeeded for a specific model. |
|
|
||
| * *Number of requests* - The number of HTTP requests that have failed or succeeded for a specific model. | ||
| * *Average response time (ms)* - The average time it takes a specific model to respond to requests. | ||
| * *CPU utilization (%)* - The percentage of the CPU's capacity that is currently being used by a specific model. |
There was a problem hiding this comment.
Suggested change
| * *CPU utilization (%)* - The percentage of the CPU's capacity that is currently being used by a specific model. | |
| * *CPU utilization (%)* - The percentage of the model deployment's CPU's limit that is currently being used by a specific model. |
| * *Number of requests* - The number of HTTP requests that have failed or succeeded for a specific model. | ||
| * *Average response time (ms)* - The average time it takes a specific model to respond to requests. | ||
| * *CPU utilization (%)* - The percentage of the CPU's capacity that is currently being used by a specific model. | ||
| * *Memory utilization (%)* - The percentage of the system's memory that is currently being used by a specific model. |
There was a problem hiding this comment.
Suggested change
| * *Memory utilization (%)* - The percentage of the system's memory that is currently being used by a specific model. | |
| * *Memory utilization (%)* - The percentage of the model deployment's memory limit that is currently being used by a specific model. |
There was a problem hiding this comment.
@VedantMahabaleshwarkar What does model deployment mean here?
There was a problem hiding this comment.
Pod if there's only 1 replica, pods if there are multiple replicas. But what I specifically mean is the Openshift Deployment that represents the model. The deployment will have resource requests and limits. The % we show is the % relative to the limit that is set in the deployment
There was a problem hiding this comment.
@VedantMahabaleshwarkar I think that model deployment might sound confusing - proposing an alternative, let me know your thoughts:
- CPU utilization (%): The percentage of the CPU limit per model replica that is currently utilized by a specific model.
- Memory utilization (%) - The percentage of memory limit per model replica that is currently utilized by a specific model.
Based my assumption on this setting:
|
Previews look good to me, just some minor suggestions as made previously |
aduquett
reviewed
Aug 8, 2024
| @@ -10,7 +10,19 @@ You can view a graph that illustrates the HTTP requests that have failed or succ | |||
| .Prerequisites | |||
| * You have installed {productname-long}. | |||
| * On the OpenShift cluster where {productname-short} is installed, user workload monitoring is enabled. | |||
| * Your cluster administrator has _not_ edited the {productname-short} dashboard configuration to hide the *Endpoint Performance* tab on the *Model Serving* page. For more information, see link:{rhoaidocshome}/html/managing_resources/customizing-the-dashboard#ref-dashboard-configuration-options_dashboard[Dashboard configuration options]. | |||
| * The following dashboard configuration options are set to their default values as shown: | |||
aduquett
reviewed
Aug 8, 2024
| @@ -0,0 +1,72 @@ | |||
| :_module-type: PROCEDURE | |||
|
|
|||
| [id="viewing-performance-metrics-for-model-server_{context}"] | |||
There was a problem hiding this comment.
Should the title and id be the same?
aduquett
reviewed
Aug 8, 2024
| ifdef::upstream[] | ||
| * If you are using specialized {productname-short} groups, you are part of the user group or admin group (for example, {odh-user-group} or {odh-admin-group}) in OpenShift. | ||
| endif::[] | ||
| * The following dashboard configuration options are set to their default values as shown: |
|
A few minor comments, but otherwise LGTM. |
syaseen-rh
force-pushed
the
RHOAIENG-4198
branch
from
b314463
to
9625679
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Initial commit for KServe performance metrics feature.
How Has This Been Tested?
Local build
Previews:
Monitoring Model Performance (Multi-model serving platform)
Monitoring Model Performance (Single-model serving platform)