[EPIC] Watsonx.ai Release Dev Tracker

[heyselbi]

• Feature Tracker: Single Model Deployment (KServe) opendatahub-community#54

Goal 1: Integration of Watsonx.ai components into ODH

Req. 1: [P0] The following components must be included out-of-the-box in RHODS at a GA support level

--> Caikit (Compositional AI Kit) serving runtime
--> TGIS (Text Generation Inference Service)
--> KServe, Service Mesh, Serverless

• [P0] Create KServe community images #2
• [P0] Update "Sync Upstream" template with tasks related to KServe #3
• [P0] Update KServe quickstart #4
• [P0] Update KServe e2e #5
• [P1] Look into potential conflict with Service Mesh integration #7
• [P0] KServe and TGIS runtime exploration #11
• Adjust Kserve's python image uid for ODH #13
• Fix image pushing with GitHub actions #27
• CPaaS onboarding of KServe #30
• Setup odh-nightlies for KServe #34
• self signed certs support to pull models #36
• Enable SNYK scan on Watson X repos #37
• Setup Openshift-CI for kserve #45
• https://github.com/opendatahub-io/odh-manifests/issues/848
• https://github.com/opendatahub-io/odh-manifests/issues/886
• Create a kserve auxillary controller to manage extra resources for LLM serving odh-model-controller#58
• Automated smoke test for caikit serving caikit-tgis-serving#7
• [RFE] support "custom" namespaces caikit-tgis-serving#24
• [SPIKE] Impact of TGIS's new req on safetensor format caikit-tgis-serving#33
• [P0] CAIKit runtime exploration caikit#1
• [P0] Configure job publish pypi for Open Data Hub caikit#2
• [P0] Demo with Kserve + Caikit + TGIS + NVIDIA GPUs caikit#5
• [P0] Demo with Kserve + Caikit + TGIS + NVIDIA GPUs with opendatahub caikit#6
• Setup operator-ci for release branch of caikit-tgis-serving caikit#8
• [P0] Create dockerfile (embedding TGIS for now) - Open Data Hub caikit-nlp#1
• [P0] Create CAIKit-nlp community images/CI for Open Data Hub caikit-nlp#2
• [SPIKE] Make ODH-model-controller able to deploy caikit+TGIS to support not KServe standard deployment models (i.e. multiple pod per runtime) caikit-nlp#4
• Caikit-nlp: sync with upstream caikit-nlp#5
• [P0] Review/validate existing dockerfile for Open Data Hub text-generation-inference#2
• [P0] Create TGIS community images/CI for Open Data Hub text-generation-inference#3
• Fix issue with TGIS loading in openshift-ci built images text-generation-inference#8
• Setup operator-ci for release branch of TGIS text-generation-inference#11

Goal 2: Seamless updates to models and Caikit

Req. 2: [P0] Users must be able to deploy an updated version of a foundation model

For example, if using a curated IBM model, when IBM releases a new model version, users must be able to deploy the updated model version. The same scenario applies to foundation models from other sources such as Hugging Face.

• [SPIKE] The system must support the ability to update the Caikit serving runtime version without impacting actively served models caikit-tgis-serving#11
• Methods for upgrading Models and Serving Runtimes #40

Req. 4: [P0] The system must support the ability to update the Caikit serving runtime version without impacting actively served models

For example, the upstream version will be updated and need to incorporate in RHODS as appropriate without impacting deployed models. A new RHODS release must not break model serving functionality.

• [P0] Users must be able to deploy an updated version of a foundation model #24

Goal 3: Monitoring & Metrics

Req. 5: [P0] Users must be able to access all applicable metrics for any deployed model

Metrics required:
--> number of inference requests over defined time period
--> Avg. response time over defined time period
--> number of successful / failed inference requests over defined time period
--> Compute utilization (CPU, GPU, Memory)

• Overall tracker for metrics: SPIKE: Investigate kserve monitoring and metrics opendatahub-community#70
  • [P0] Configure KServe runtime in monitoring #31
  • [P0] Configure KServe controller for monitoring #32
  • [P0] Verify which metrics CAIKit/TGIS produce caikit#3

Goal 4: Support for HF models

Req. 3: [P0] The product must support the ability to deploy foundation models from Hugging Face using OOTB capabilities.

To be clear, we’re not looking to support the actual models themselves, but rather the ability to deploy/serve the models. If a customer has an issue with the actual model, that is outside the support scope.

Goal 5: HTTP and gRPC endpoints

Req. 6: [P0] Enable users to create & access http and grpc endpoints for model serving routes

• [P0] SPIKE: making sure the long running connection works (http/grpc streaming) #33