Skip to content

Commit

Permalink
Add Helm monitoring option documentation
Browse files Browse the repository at this point in the history 
And refactor + fix HPA instructions (HPA setting is not global).

Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com>
  • Loading branch information
@eero-t
eero-t committed Oct 18, 2024
1 parent 61b3ba9 commit 6f7e230
Show file tree
Hide file tree
Showing 11 changed files with 127 additions and 67 deletions.
69 changes: 14 additions & 55 deletions helm-charts/HPA.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
- [Introduction](#introduction)
- [Pre-conditions](#pre-conditions)
- [Resource requests](#resource-requests)
- [Prometheus](#prometheus)
- [Prometheus metrics](#prometheus-metrics)
- [Prometheus-adapter](#prometheus-adapter)
- [Gotchas](#gotchas)
- [Enable HPA](#enable-hpa)
- [Install](#install)
Expand All @@ -14,7 +15,7 @@

## Introduction

`horizontalPodAutoscaler` option enables HPA scaling for the TGI and TEI inferencing deployments:
`horizontalPodAutoscaler` option enables HPA scaling for relevant service components:
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/).
Expand All @@ -41,24 +42,21 @@ small requests would be an issue:
that provide further isolation
- Containers can become non-functional when their actual resource usage crosses the specified limits

### Prometheus
### Prometheus metrics

If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus)
yet, it SHOULD be be installed before enabling HPA, e.g. by using a Helm chart for it:
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Autoscaling requires k8s Prometheus installation and monitoring to be enabled in the top level chart.
See [monitoring instructions](monitoring.md) for details.

Prometheus-adapter is also needed, to provide k8s custom metrics based on collected TGI / TEI metrics:
### Prometheus-adapter

Prometheus-adapter is also needed, to provide k8s custom metrics based on collected service metrics:
https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter

To install (older versions) of them:
Install adapter after installing Prometheus:

```console
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ prom_ns=monitoring # namespace for Prometheus/-adapter
$ kubectl create ns $prom_ns
$ helm install prometheus-stack prometheus-community/kube-prometheus-stack --version 55.5.2 -n $prom_ns
$ kubectl get services -n $prom_ns
$ kubectl get svc -n $prom_ns
$ helm install prometheus-adapter prometheus-community/prometheus-adapter --version 4.10.0 -n $prom_ns \
--set prometheus.url=http://prometheus-stack-kube-prom-prometheus.$prom_ns.svc \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
Expand All @@ -80,9 +78,6 @@ Why HPA is opt-in:
Prometheus-operator and -adapter are missing support needed to automate that
- Top level chart name needs to conform to Prometheus metric naming conventions,
as it is also used as a metric name prefix (with dashes converted to underscores)
- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml)
for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is
asked to install OPEA services to some other namespace, those rules need to be updated accordingly
- Unless pod resource requests, affinity rules, scheduling topology constraints and/or cluster NRI
policies are used to better isolate service inferencing pods from each other, instances
scaled up on same node may never get to ready state
Expand Down Expand Up @@ -143,46 +138,10 @@ $ kubectl -n $prom_ns delete $(kubectl -n $prom_ns get pod --selector $selector

## Verify

To verify that horizontalPodAutoscaler options work, it's better to check that both metrics
from the inferencing services, and HPA rules using custom metrics generated from them, do work.

(Names of the object names depend on whether Prometheus was installed from manifests, or Helm,
and the release name given for its Helm install.)

Check installed Prometheus service names:

```console
$ prom_ns=monitoring # Prometheus/-adapter namespace
$ kubectl -n $prom_ns get svc
```

Use service name matching your Prometheus installation:

```console
$ prom_svc=prometheus-stack-kube-prom-prometheus # Metrics service
```

Verify Prometheus found metric endpoints for chart services, i.e. last number on `curl` output is non-zero:

```console
$ chart=chatqna # OPEA chart release name
$ prom_url=http://$(kubectl -n $prom_ns get -o jsonpath="{.spec.clusterIP}:{.spec.ports[0].port}" svc/$prom_svc)
$ curl --no-progress-meter $prom_url/metrics | grep scrape_pool_targets.*$chart
```

**NOTE**: TGI and TEI inferencing services provide metrics endpoint only after they've processed
their first request, and reranking service will be used only after context data has been uploaded!

Check that both Prometheus metrics required from TGI are available:

```console
$ for m in sum count; do
curl --no-progress-meter $prom_url/api/v1/query? \
--data-urlencode query=tgi_request_inference_duration_$m'{service="'$chart'-tgi"}' | jq;
done | grep __name__
```
After [verifying that service metrics work](monitoring.md#verify),
one can verify that HPA rules can access custom metrics based on them.

PrometheusAdapter lists corresponding TGI and/or TEI custom metrics (prefixed with chart name):
Verify that there are (TGI and/or TEI) custom metrics prefixed with chart name:

```console
$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .resources[].name
Expand Down
7 changes: 6 additions & 1 deletion helm-charts/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This directory contains Helm charts for [GenAIComps](https://github.com/opea-pro
- [From Source Code](#from-source-code)
- [Using Helm Charts repository](#using-helm-charts-repository)
- [Helm Charts Options](#helm-charts-options)
- [Using HPA (autoscaling)](#using-hpa-autoscaling)
- [Using Persistent Volume](#using-persistent-volume)
- [Using Private Docker Hub](#using-private-docker-hub)
- [Generate manifests from Helm Charts](#generate-manifests-from-helm-charts)
Expand Down Expand Up @@ -88,9 +89,13 @@ There are global options (which should be shared across all components of a work
| global | http_proxy https_proxy no_proxy | Proxy settings. If you are running the workloads behind the proxy, you'll have to add your proxy settings here. |
| global | modelUsePVC | The PersistentVolumeClaim you want to use as HuggingFace hub cache. Default "" means not using PVC. Only one of modelUsePVC/modelUseHostPath can be set. |
| global | modelUseHostPath | If you don't have Persistent Volume in your k8s cluster and want to use local directory as HuggingFace hub cache, set modelUseHostPath to your local directory name. Note that this can't share across nodes. Default "". Only one of modelUsePVC/modelUseHostPath can be set. |
| chatqna | horizontalPodAutoscaler.enabled | Enable HPA autoscaling for TGI and TEI service deployments based on metrics they provide. See [Pre-conditions](HPA.md#pre-conditions) and [Gotchas](HPA.md#gotchas) before enabling! |
| global | monitoring | Enable monitoring for (ChatQnA) service components. See [Pre-conditions](monitoring.md#pre-conditions) before enabling! |
| tgi | LLM_MODEL_ID | The model id you want to use for tgi server. Default "Intel/neural-chat-7b-v3-3". |

## Using HPA (autoscaling)

See [HPA instructions](HPA.md) on how to enable horizontal pod autoscaling for service components, based on their usage metrics.

## Using Persistent Volume

It's common to use Persistent Volume (PV) for model caches (HuggingFace hub cache) in a production k8s cluster. PersistentVolumeClaim (PVC) can be passed to containers, but it's the user's responsibility to create the PVC depending on your k8s cluster's capability.
Expand Down
16 changes: 8 additions & 8 deletions helm-charts/chatqna/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --

### IMPORTANT NOTE

1. Make sure your `MODELDIR` exists on the node where your workload is schedueled so you can cache the downloaded model for next time use. Otherwise, set `global.modelUseHostPath` to 'null' if you don't want to cache the model.
1. Make sure your `MODELDIR` exists on the node where your workload is scheduled so you can cache the downloaded model for next time use. Otherwise, set `global.modelUseHostPath` to 'null' if you don't want to cache the model.

## Verify

Expand Down Expand Up @@ -71,13 +71,13 @@ Open a browser to access `http://<k8s-node-ip-address>:${port}` to play with the

## Values

| Key | Type | Default | Description |
| -------------------------------------- | ------ | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| image.repository | string | `"opea/chatqna"` | |
| service.port | string | `"8888"` | |
| tgi.LLM_MODEL_ID | string | `"Intel/neural-chat-7b-v3-3"` | Models id from https://huggingface.co/, or predownloaded model directory |
| global.horizontalPodAutoscaler.enabled | bop; | false | HPA autoscaling for the TGI and TEI service deployments based on metrics they provide. See HPA section in ../README.md before enabling! |
| Key | Type | Default | Description |
| ----------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------- |
| image.repository | string | `"opea/chatqna"` | |
| service.port | string | `"8888"` | |
| tgi.LLM_MODEL_ID | string | `"Intel/neural-chat-7b-v3-3"` | Models id from https://huggingface.co/, or predownloaded model directory |
| global.monitoring | bop; | false | Enable usage metrics for the service components. See ../monitoring.md before enabling! |

## Troubleshooting

If you encount any issues, please refer to [ChatQnA Troubleshooting](troubleshooting.md)
If you encounter any issues, please refer to [ChatQnA Troubleshooting](troubleshooting.md)
1 change: 1 addition & 0 deletions helm-charts/common/embedding-usvc/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,4 @@ curl http://localhost:6000/v1/embeddings \
| image.repository | string | `"opea/embedding-tei"` | |
| service.port | string | `"6000"` | |
| TEI_EMBEDDING_ENDPOINT | string | `""` | |
| global.monitoring | bop; | false | |
1 change: 1 addition & 0 deletions helm-charts/common/llm-uservice/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,4 @@ curl http://localhost:9000/v1/chat/completions \
| image.repository | string | `"opea/llm-tgi"` | |
| service.port | string | `"9000"` | |
| TGI_LLM_ENDPOINT | string | `""` | LLM endpoint |
| global.monitoring | bop; | false | Service usage metrics |
1 change: 1 addition & 0 deletions helm-charts/common/reranking-usvc/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,4 @@ curl http://localhost:8000/v1/reranking \
| image.repository | string | `"opea/reranking-tgi"` | |
| TEI_RERANKING_ENDPOINT | string | `""` | |
| service.port | string | `"8000"` | |
| global.monitoring | bop; | false | |
1 change: 1 addition & 0 deletions helm-charts/common/retriever-usvc/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ curl http://localhost:7000/v1/retrieval \
| service.port | string | `"7000"` | |
| REDIS_URL | string | `""` | |
| TEI_EMBEDDING_ENDPOINT | string | `""` | |
| global.monitoring | bop; | false | |

## Milvus support

Expand Down
3 changes: 2 additions & 1 deletion helm-charts/common/tei/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,5 @@ curl http://localhost:2081/embed -X POST -d '{"inputs":"What is Deep Learning?"}
| global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, tei will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. |
| image.repository | string | `"ghcr.io/huggingface/text-embeddings-inference"` | |
| image.tag | string | `"cpu-1.5"` | |
| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA section](../../HPA.md) before enabling! |
| horizontalPodAutoscaler.enabled | bop; | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA instructions](../../HPA.md) before enabling! |
| global.monitoring | bop; | false | Enable usage metrics for the service. Required for HPA. See [monitoring instructions](../../monitoring.md) before enabling! |
3 changes: 2 additions & 1 deletion helm-charts/common/teirerank/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ curl http://localhost:2082/rerank \
| global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, teirerank will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. |
| image.repository | string | `"ghcr.io/huggingface/text-embeddings-inference"` | |
| image.tag | string | `"cpu-1.5"` | |
| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA section](../../HPA.md) before enabling! |
| horizontalPodAutoscaler.enabled | bop; | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA instructions](../../HPA.md) before enabling! |
| global.monitoring | bop; | false | Enable usage metrics for the service. Required for HPA. See [monitoring instructions](../../monitoring.md) before enabling! |
3 changes: 2 additions & 1 deletion helm-charts/common/tgi/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,5 @@ curl http://localhost:2080/generate \
| global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, tgi will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. |
| image.repository | string | `"ghcr.io/huggingface/text-generation-inference"` | |
| image.tag | string | `"1.4"` | |
| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA section](../../HPA.md) before enabling! |
| horizontalPodAutoscaler.enabled | bop; | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See [HPA instructions](../../HPA.md) before enabling! |
| global.monitoring | bop; | false | Enable usage metrics for the service. Required for HPA. See [monitoring instructions](../../monitoring.md) before enabling! |
Loading

0 comments on commit 6f7e230

Please sign in to comment.