diff --git a/helm-charts/README.md b/helm-charts/README.md index 2bec0cf2..c4eef858 100644 --- a/helm-charts/README.md +++ b/helm-charts/README.md @@ -9,6 +9,10 @@ This directory contains helm charts for [GenAIComps](https://github.com/opea-pro - [Components](#components) - [How to deploy with helm charts](#deploy-with-helm-charts) - [Helm Charts Options](#helm-charts-options) +- [HorizontalPodAutoscaler (HPA) support](#horizontalpodautoscaler-hpa-support) + - [Pre-conditions](#pre-conditions) + - [Gotchas](#gotchas) + - [Verify HPA metrics](#verify-hpa-metrics) - [Using Persistent Volume](#using-persistent-volume) - [Using Private Docker Hub](#using-private-docker-hub) - [Helm Charts repository](#helm-chart-repository) @@ -62,8 +66,71 @@ There are global options(which should be shared across all components of a workl | global | http_proxy https_proxy no_proxy | Proxy settings. If you are running the workloads behind the proxy, you'll have to add your proxy settings here. | | global | modelUsePVC | The PersistentVolumeClaim you want to use as huggingface hub cache. Default "" means not using PVC. Only one of modelUsePVC/modelUseHostPath can be set. | | global | modelUseHostPath | If you don't have Persistent Volume in your k8s cluster and want to use local directory as huggingface hub cache, set modelUseHostPath to your local directory name. Note that this can't share across nodes. Default "". Only one of modelUsePVC/modelUseHostPath can be set. | +| global | horizontalPodAutoscaler.enabled | Enable HPA autoscaling for TGI and TEI service deployments based on metrics they provide. See #pre-conditions and #gotchas before enabling! | | tgi | LLM_MODEL_ID | The model id you want to use for tgi server. Default "Intel/neural-chat-7b-v3-3". | +## HorizontalPodAutoscaler (HPA) support + +`horizontalPodAutoscaler` option enables HPA scaling for the TGI and TEI inferencing deployments: +https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ + +Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/). + +### Pre-conditions + +If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus) +yet, it SHOULD be be installed before enabling HPA, e.g. by using: +https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack + +Enabling HPA in top-level Helm chart (e.g. `chatqna`), overwrites cluster's current _PrometheusAdapter_ +configuration with relevant custom metric queries. If that has queries you wish to retain, _or_ HPA is +otherwise enabled only in TGI or TEI subchart(s), you need add relevat queries to _PrometheusAdapter_ +configuration _manually_ (e.g. from `chatqna` custom metrics Helm template). + +### Gotchas + +Why HPA is opt-in: + +- Enabling (top level) chart `horizontalPodAutoscaler` option will _overwrite_ cluster's current + `PrometheusAdapter` configuration with its own custom metrics configuration. + Take copy of the existing one before install, if that matters: + `kubectl -n monitoring get cm/adapter-config -o yaml > adapter-config.yaml` +- `PrometheusAdapter` needs to be restarted after install, for it to read the new configuration: + `ns=monitoring; kubectl -n $ns delete $(kubectl -n $ns get pod --selector app.kubernetes.io/name=prometheus-adapter -o name)` +- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml) + for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is + asked to install OPEA services to some other namespace, those rules need to be updated accordingly +- Current HPA rules are examples for Xeon, for efficient scaling they need to be fine-tuned for given setup + performance (underlying HW, used models and data types, OPEA version etc) + +### Verify HPA metrics + +To verify that metrics required by horizontalPodAutoscaler option work, check following... + +Prometheus has found the metric endpoints, i.e. last number on `curl` output is non-zero: + +```console +chart=chatqna; # OPEA services prefix +ns=monitoring; # Prometheus namespace +prom_url=http://$(kubectl -n $ns get -o jsonpath="{.spec.clusterIP}:{.spec.ports[0].port}" svc/prometheus-k8s); +curl --no-progress-meter $prom_url/metrics | grep scrape_pool_targets.*$chart +``` + +**NOTE**: TGI and TEI inferencing services provide metrics endpoint only after they've processed their first request! + +PrometheusAdapter lists TGI and/or TGI custom metrics (`te_*` / `tgi_*`): + +```console +kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .resources[].name +``` + +HPA rules list valid (not ``) TARGET values for service deployments: + +```console +ns=default; # OPEA namespace +kubectl -n $ns get hpa +``` + ## Using Persistent Volume It's common to use Persistent Volume(PV) for model caches(huggingface hub cache) in a production k8s cluster. We support to pass the PersistentVolumeClaim(PVC) to containers, but it's the user's responsibility to create the PVC depending on your k8s cluster's capability. diff --git a/helm-charts/chatqna/README.md b/helm-charts/chatqna/README.md index 28191e6f..64f001eb 100644 --- a/helm-charts/chatqna/README.md +++ b/helm-charts/chatqna/README.md @@ -34,35 +34,6 @@ helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} -- 1. Make sure your `MODELDIR` exists on the node where your workload is schedueled so you can cache the downloaded model for next time use. Otherwise, set `global.modelUseHostPath` to 'null' if you don't want to cache the model. -## HorizontalPodAutoscaler (HPA) support - -`horizontalPodAutoscaler` option enables HPA scaling for the TGI and TEI inferencing deployments: -https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ - -Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/). - -### Pre-conditions - -If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus) -yet, it SHOULD be be installed before enabling HPA, e.g. by using: -https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - -### Gotchas - -Why HPA is opt-in: - -- Enabling chart `horizontalPodAutoscaler` option will _overwrite_ cluster's current - `PrometheusAdapter` configuration with its own custom metrics configuration. - Take copy of the existing one before install, if that matters: - `kubectl -n monitoring get cm/adapter-config -o yaml > adapter-config.yaml` -- `PrometheusAdapter` needs to be restarted after install, for it to read the new configuration: - `ns=monitoring; kubectl -n $ns delete $(kubectl -n $ns get pod --selector app.kubernetes.io/name=prometheus-adapter -o name)` -- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml) - for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is - asked to install OPEA services to some other namespace, those rules need to be updated accordingly -- Provided HPA rules are examples for Xeon, for efficient scaling they need to be fine-tuned for given setup - (underlying HW, used models, OPEA version etc) - ## Verify To verify the installation, run the command `kubectl get pod` to make sure all pods are running. @@ -112,9 +83,9 @@ Access `http://localhost:5174` to play with the ChatQnA workload through UI. ## Values -| Key | Type | Default | Description | -| -------------------------------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| image.repository | string | `"opea/chatqna"` | | -| service.port | string | `"8888"` | | -| tgi.LLM_MODEL_ID | string | `"Intel/neural-chat-7b-v3-3"` | Models id from https://huggingface.co/, or predownloaded model directory | -| global.horizontalPodAutoscaler.enabled | bop; | false | HPA autoscaling for the TGI and TEI service deployments based on metrics they provide. See #pre-conditions and #gotchas before enabling! (If one doesn't want one of them to be scaled, given service `maxReplicas` can be set to `1`) | +| Key | Type | Default | Description | +| -------------------------------------- | ------ | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| image.repository | string | `"opea/chatqna"` | | +| service.port | string | `"8888"` | | +| tgi.LLM_MODEL_ID | string | `"Intel/neural-chat-7b-v3-3"` | Models id from https://huggingface.co/, or predownloaded model directory | +| global.horizontalPodAutoscaler.enabled | bop; | false | HPA autoscaling for the TGI and TEI service deployments based on metrics they provide. See HPA section in ../README.md before enabling! | diff --git a/helm-charts/common/tei/README.md b/helm-charts/common/tei/README.md index 095e33f9..9d9817ea 100644 --- a/helm-charts/common/tei/README.md +++ b/helm-charts/common/tei/README.md @@ -21,38 +21,6 @@ MODELDIR=/mnt/opea-models MODELNAME="/data/BAAI/bge-base-en-v1.5" -## HorizontalPodAutoscaler (HPA) support - -`horizontalPodAutoscaler` option enables HPA scaling for the deployment: -https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ - -Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/). - -### Pre-conditions - -If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus) -yet, it SHOULD be be installed before enabling HPA, e.g. by using: -https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - -`horizontalPodAutoscaler` enabled in top level Helm chart depending on this component (e.g. `chatqna`), -so that relevant custom metric queries are configured for PrometheusAdapter. - -### Gotchas - -Why HPA is opt-in: - -- Enabling chart `horizontalPodAutoscaler` option will _overwrite_ cluster's current - `PrometheusAdapter` configuration with its own custom metrics configuration. - Take copy of the existing one before install, if that matters: - `kubectl -n monitoring get cm/adapter-config -o yaml > adapter-config.yaml` -- `PrometheusAdapter` needs to be restarted after install, for it to read the new configuration: - `ns=monitoring; kubectl -n $ns delete $(kubectl -n $ns get pod --selector app.kubernetes.io/name=prometheus-adapter -o name)` -- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml) - for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is - asked to install OPEA services to some other namespace, those rules need to be updated accordingly -- Provided HPA rules are examples for Xeon, for efficient scaling they need to be fine-tuned for given setup - (underlying HW, used models, OPEA version etc) - ## Verify To verify the installation, run the command `kubectl get pod` to make sure all pods are runinng. @@ -65,35 +33,6 @@ Open another terminal and run the following command to verify the service if wor curl http://localhost:2081/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json' ``` -### Verify HPA metrics - -To verify that metrics required by horizontalPodAutoscaler option work, check that: - -Prometheus has found the metric endpoints, i.e. last number on the line is non-zero: - -```console -prom_url=http://$(kubectl -n monitoring get -o jsonpath="{.spec.clusterIP}:{.spec.ports[0].port}" svc/prometheus-k8s) -curl --no-progress-meter $prom_url/metrics | grep scrape_pool_targets.*tei -``` - -Prometheus adapter provides custom metrics for their data: - -```console -kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .resources[].name -``` - -And those custom metrics have valid values for HPA rules: - -```console -ns=default; # OPEA namespace -url=/apis/custom.metrics.k8s.io/v1beta1; -for m in $(kubectl get --raw $url | jq .resources[].name | cut -d/ -f2 | sort -u | tr -d '"'); do - kubectl get --raw $url/namespaces/$ns/metrics/$m | jq; -done | grep -e metricName -e value -``` - -NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they've processed their first request! - ## Values | Key | Type | Default | Description | @@ -102,4 +41,4 @@ NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they' | global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, tei will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. | | image.repository | string | `"ghcr.io/huggingface/text-embeddings-inference"` | | | image.tag | string | `"cpu-1.5"` | | -| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployments based on metrics it provides. See #pre-conditions and #gotchas before enabling! | +| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See HPA section in ../../README.md before enabling! | diff --git a/helm-charts/common/teirerank/README.md b/helm-charts/common/teirerank/README.md index e1c8d216..d445364f 100644 --- a/helm-charts/common/teirerank/README.md +++ b/helm-charts/common/teirerank/README.md @@ -21,38 +21,6 @@ MODELDIR=/mnt/opea-models MODELNAME="/data/BAAI/bge-reranker-base" -## HorizontalPodAutoscaler (HPA) support - -`horizontalPodAutoscaler` option enables HPA scaling for the deployment: -https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ - -Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/). - -### Pre-conditions - -If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus) -yet, it SHOULD be be installed before enabling HPA, e.g. by using: -https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - -`horizontalPodAutoscaler` enabled in top level Helm chart depending on this component (e.g. `chatqna`), -so that relevant custom metric queries are configured for PrometheusAdapter. - -### Gotchas - -Why HPA is opt-in: - -- Enabling chart `horizontalPodAutoscaler` option will _overwrite_ cluster's current - `PrometheusAdapter` configuration with its own custom metrics configuration. - Take copy of the existing one before install, if that matters: - `kubectl -n monitoring get cm/adapter-config -o yaml > adapter-config.yaml` -- `PrometheusAdapter` needs to be restarted after install, for it to read the new configuration: - `ns=monitoring; kubectl -n $ns delete $(kubectl -n $ns get pod --selector app.kubernetes.io/name=prometheus-adapter -o name)` -- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml) - for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is - asked to install OPEA services to some other namespace, those rules need to be updated accordingly -- Provided HPA rules are examples for Xeon, for efficient scaling they need to be fine-tuned for given setup - (underlying HW, used models, OPEA version etc) - ## Verify To verify the installation, run the command `kubectl get pod` to make sure all pods are runinng. @@ -68,35 +36,6 @@ curl http://localhost:2082/rerank \ -H 'Content-Type: application/json' ``` -### Verify HPA metrics - -To verify that metrics required by horizontalPodAutoscaler option work, check that: - -Prometheus has found the metric endpoints, i.e. last number on the line is non-zero: - -```console -prom_url=http://$(kubectl -n monitoring get -o jsonpath="{.spec.clusterIP}:{.spec.ports[0].port}" svc/prometheus-k8s) -curl --no-progress-meter $prom_url/metrics | grep scrape_pool_targets.*rerank -``` - -Prometheus adapter provides custom metrics for their data: - -```console -kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .resources[].name -``` - -And those custom metrics have valid values for HPA rules: - -```console -ns=default; # OPEA namespace -url=/apis/custom.metrics.k8s.io/v1beta1; -for m in $(kubectl get --raw $url | jq .resources[].name | cut -d/ -f2 | sort -u | tr -d '"'); do - kubectl get --raw $url/namespaces/$ns/metrics/$m | jq; -done | grep -e metricName -e value -``` - -NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they've processed their first request! - ## Values | Key | Type | Default | Description | @@ -105,4 +44,4 @@ NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they' | global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, teirerank will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. | | image.repository | string | `"ghcr.io/huggingface/text-embeddings-inference"` | | | image.tag | string | `"cpu-1.5"` | | -| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployments based on metrics it provides. See #pre-conditions and #gotchas before enabling! | +| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See HPA section in ../../README.md before enabling! | diff --git a/helm-charts/common/tgi/README.md b/helm-charts/common/tgi/README.md index 23a9e664..0100378f 100644 --- a/helm-charts/common/tgi/README.md +++ b/helm-charts/common/tgi/README.md @@ -24,38 +24,6 @@ MODELDIR=/mnt/opea-models MODELNAME="/data/models--bigscience--bloom-560m" -## HorizontalPodAutoscaler (HPA) support - -`horizontalPodAutoscaler` option enables HPA scaling for the deployment: -https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ - -Autoscaling is based on custom application metrics provided through [Prometheus](https://prometheus.io/). - -### Pre-conditions - -If cluster does not run [Prometheus operator](https://github.com/prometheus-operator/kube-prometheus) -yet, it SHOULD be be installed before enabling HPA, e.g. by using: -https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - -`horizontalPodAutoscaler` enabled in top level Helm chart depending on this component (e.g. `chatqna`), -so that relevant custom metric queries are configured for PrometheusAdapter. - -### Gotchas - -Why HPA is opt-in: - -- Enabling chart `horizontalPodAutoscaler` option will _overwrite_ cluster's current - `PrometheusAdapter` configuration with its own custom metrics configuration. - Take copy of the existing one before install, if that matters: - `kubectl -n monitoring get cm/adapter-config -o yaml > adapter-config.yaml` -- `PrometheusAdapter` needs to be restarted after install, for it to read the new configuration: - `ns=monitoring; kubectl -n $ns delete $(kubectl -n $ns get pod --selector app.kubernetes.io/name=prometheus-adapter -o name)` -- By default Prometheus adds [k8s RBAC rules](https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml) - for accessing metrics from `default`, `kube-system` and `monitoring` namespaces. If Helm is - asked to install OPEA services to some other namespace, those rules need to be updated accordingly -- Provided HPA rules are examples for Xeon, for efficient scaling they need to be fine-tuned for given setup - (underlying HW, used models, OPEA version etc) - ## Verify To verify the installation, run the command `kubectl get pod` to make sure all pods are runinng. @@ -71,35 +39,6 @@ curl http://localhost:2080/generate \ -H 'Content-Type: application/json' ``` -### Verify HPA metrics - -To verify that metrics required by horizontalPodAutoscaler option work, check that: - -Prometheus has found the metric endpoints, i.e. last number on the line is non-zero: - -```console -prom_url=http://$(kubectl -n monitoring get -o jsonpath="{.spec.clusterIP}:{.spec.ports[0].port}" svc/prometheus-k8s) -curl --no-progress-meter $prom_url/metrics | grep scrape_pool_targets.*tgi -``` - -Prometheus adapter provides custom metrics for their data: - -```console -kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .resources[].name -``` - -And those custom metrics have valid values for HPA rules: - -```console -ns=default; # OPEA namespace -url=/apis/custom.metrics.k8s.io/v1beta1; -for m in $(kubectl get --raw $url | jq .resources[].name | cut -d/ -f2 | sort -u | tr -d '"'); do - kubectl get --raw $url/namespaces/$ns/metrics/$m | jq; -done | grep -e metricName -e value -``` - -NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they've processed their first request! - ## Values | Key | Type | Default | Description | @@ -109,4 +48,4 @@ NOTE: HuggingFace TGI and TEI services provide metrics endpoint only after they' | global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, tgi will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory. Set this to null/empty will force it to download model. | | image.repository | string | `"ghcr.io/huggingface/text-generation-inference"` | | | image.tag | string | `"1.4"` | | -| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployments based on metrics it provides. See #pre-conditions and #gotchas before enabling! | +| horizontalPodAutoscaler.enabled | bool | false | Enable HPA autoscaling for the service deployment based on metrics it provides. See HPA section in ../../README.md before enabling! |