Split caikit-tgis into separate containers

[dtrifiro]

Description

• add Dockerfile for caikit-nlp with tgis backend
• update docs example (InferenceService+ServingRuntime)
• remove smoketest (this breaks with the new caikit-only image

How Has This Been Tested?

This can be tested by running the provided example in either a local kserve deployment or openshift:

This example uses a pvc/pvc, but the model can be also stored in s3 or other object storage

1. Create a PV/PVC

apiVersion: v1
kind: PersistentVolume
metadata:
  name: caikit-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/models/"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: caikit-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

Download a model and convert it to caikit format (note: this requires git-lfs to be installed and configured

git clone https://huggingface.co/google/flan-t5-small
python -c 'import caikit_nlp; \
model = caikit_nlp.text_generation.TextGeneration.bootstrap("flan-t5-small"); \
model.save("flan-t5-small-caikit")'

2. Copy the model to the pv

3. Edit docs/caikit-isvc.yaml and make storageUri point to the model:
   storageUri: pvc://caikit-claim/flan-t5-small-caikit/

3 Apply the manifests,

oc apply -f docs/caikit-isvc.yaml -f docs/caikit-servingruntime.yaml

4. After port forwarding to the inference service, it should be able to query the grpc endpoint

oc port-forward deployments/caikit-example-isvc-predictor-00001-deployment 8085 &>/dev/null &
grpcurl -plaintext -d '{"text": "At what temperature does Nitrogen boil?"}' -H "mm-model-id: flan-t5-small-caikit" 127.0.0.1:8085 caikit.runtime.Nlp.NlpService/TextGenerationTaskPredict

Verify that the inference is running on the tgis instance by looking at the logs of the transformer container:

oc logs deployments/caikit-example-isvc-predictor-00001-deployment -c transformer-container -f     

In a subsequent PR, I'll provide an automatic method of testing this setup, either via docker-compose and/or by deploying these manifests.

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[Xaenalt]

Left a few small ones on the Dockerfile and the ServingRuntime, but overall looking great!

[Xaenalt]

Overall, PR is looking great, great job!

[Xaenalt]

Added a few more notes to my comments (message here for timestamp tracking)

[Xaenalt]

Great work!

[Xaenalt]

/hold

[dtrifiro]

Updated the images/tags names as discussed with @Xaenalt

[Xaenalt]

/unhold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtrifiro, Xaenalt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Xaenalt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kpouget]

do you guys know in which version of RHOAI this will land?

[Xaenalt]

1.36 is the official release, you can use it ahead of time from the ODH repo

[kpouget]

@Xaenalt , actually, it's this image that I'm using at the moment: quay.io/opendatahub/caikit-tgis-serving:stable
will it be updated automatically following this merge? 🤔

[dtrifiro]

@Xaenalt , actually, it's this image that I'm using at the moment: quay.io/opendatahub/caikit-tgis-serving:stable will it be updated automatically following this merge? 🤔

yep

[bdattoma]

do we need to update the KServe CRD as well? currently we ship v1beta1 @dtrifiro @Xaenalt

[Xaenalt]

I think we want to align this with the current KServe version, so I think we'd want this to be v1beta1