Cannot deploy Kserve and Modelmesh together

[VaishnaviHire]

Describe the bug
Kserve manifests and Modelmesh manifests cannot be deployed together. There is a conflict when deploying Inferenceservice CRD

Steps To Reproduce
Steps to reproduce the behavior:

1. Deploy ODH operator
2. Deploy Kfdef with modelmesh and kserve

Expected behavior
Kserve and modelmesh should be deployed together

Workaround (if any)
Update kserve manifest to match the version of modelmesh manifests.

Open Data Hub Version
Please attach relevant kfdef manifest if applicable

OpenShift Version
Version #:
Provider (Baremetal, OpenStack, RHV, AWS, OKD, CodeReady Containers, ...):

Additional context
Add any other context about the problem here.

[VaishnaviHire]

/cc @Jooho

Created this bug for reference.

[LaVLaS]

Transferring this to odh-operator issue board for tracking before opendatahub-io/opendatahub-community/issues/77 is complete

[heyselbi]

This issue has been addressed in ODH 1.11 and RHODS 1.34. It can be closed (I don't have permissions to close the issue).

[zdtsw]

This issue has been addressed in ODH 1.11 and RHODS 1.34. It can be closed (I don't have permissions to close the issue).

Does it mean, in RHODS 2.4, we can have both kserve and modelmesh components enabled at the same time?

[bdattoma]

@zdtsw according to opendatahub-io/odh-model-controller#87, this should be solved.

I checked on a cluster with 2.4, and in fact I was able to deploy a kserve model and a modelmesh one in the same cluster at the same moment.
However, when I disabled KServe, MM wasn't able to deploy models anymore bcos of kserve webhook

Internal error occurred: failed calling webhook "inferenceservice.kserve-webhook-server.defaulter": failed to call webhook: Post "https://kserve-webhook-server-service.redhat-ods-applications.svc:443/mutate-serving-kserve-io-v1beta1-inferenceservice?timeout=10s": service "kserve-webhook-server-service" not found

(same error which was reported by Wen in #508)
@heyselbi @VedantMahabaleshwarkar

[VedantMahabaleshwarkar]

@bdattoma opendatahub-io/odh-model-controller#87 this particular issue is separate from the webhook/CRD mismatch error.
We will take a look at the webhook/CRD error

[VedantMahabaleshwarkar]

@bdattoma @zdtsw

The above error is because when Kserve is removed, the MutatingWebhookConfiguration "inferenceservice.serving.kserve.io" is not removed by the operator.
I was able to successfully replicate the issue. I can also see that it has the correct labels

  labels:
    app.kubernetes.io/part-of: kserve
    app.opendatahub.io/kserve: 'true'

AFAIK, these labels are what the operator uses to clean up resources when a component is removed. Not sure why this isn't getting cleaned up.

[zdtsw]

@bdattoma @zdtsw

The above error is because when Kserve is removed, the MutatingWebhookConfiguration "inferenceservice.serving.kserve.io" is not removed by the operator. I was able to successfully replicate the issue. I can also see that it has the correct labels

  labels:
    app.kubernetes.io/part-of: kserve
    app.opendatahub.io/kserve: 'true'

AFAIK, these labels are what the operator uses to clean up resources when a component is removed. Not sure why this isn't getting cleaned up.

I think it is related to the fix by this PR #669
we had a bug which did not clean up all the left-over resources by DSC

[bdattoma]

understood @VedantMahabaleshwarkar @zdtsw . Could we target the resolution of this issue for the very next release? 2.5 then.

Anyways, @VedantMahabaleshwarkar I wonder why the webhook doesn't bother MM when kserve is enabled but it bothers when it is disabled. Does it mean that the KServe webhook is always called, even when we deploy a MM model?

[zdtsw]

understood @VedantMahabaleshwarkar @zdtsw . Could we target the resolution of this issue for the very next release? 2.5 then.

Anyways, @VedantMahabaleshwarkar I wonder why the webhook doesn't bother MM when kserve is enabled but it bothers when it is disabled. Does it mean that the KServe webhook is always called, even when we deploy a MM model?

sure, we will have the fix from our side in asap and have some local test first.

[bdattoma]

should this ticket marked as completed?

[VedantMahabaleshwarkar]

If this issue is going to track the bug related to the ODH operator not deleting the MutatingWebhookConfiguration , then I'd say it can be closed.
The remaining issue with installing Kserve and ModelMesh together related to the CRD conflicts is being tracked here : opendatahub-io/kserve#121