Seldon Core Operator Restricted to Single Namespace

[axsaucedo]

This task encompasses the functionality to deploy the Seldon Core Operator restricted to a single namespace. Our operator is built using Kubebuilder which by default was using Cluster-wide roles, however since we upgraded to Kubebuilder 2.0 we are now able to use normal roles, which would enable us to have more granular restrictions. This is also something that has been requested previously.

Related issues include #670, #890

[axsaucedo]

@phsiao @lennon310 it would be great to hear your thoughts, especially as @cliveseldon mentioned that you currnetly have a walkaround that allows you to restrict Seldon Core to a specific set of namespaces

[axsaucedo]

For context, this was discussed on the wider context around rolling updates, and how we will be aiming to support versions of Seldon. For more information on some notes, you can check out #890 (comment)

[adriangonz]

@cliveseldon do you have any ideas on what would be the best way to differentiate between namespace vs cluster installations? Is this something that can be handled through helm?

[ukclivecox]

So far I have just started the Kustomize change so it will be a different overlay namespaced
For helm it will be a setting in the values file.

[phsiao]

Some initial thought.

There are a few components that we need to stage changes in order for a meaningful test, CRD, webhook, and the operator. The operator is probably the easiest to contain, while the other two may need some thought.

An approach that I have seen that works is some form of "instance id" annotation, where if an instance of CRD is annotated with that, only the matching operator would process it. I think it might be possible to take the same approach for webhook.

[ukclivecox]

Thanks @phsiao
I think webhooks can be scoped to namespaces.
For CRDs I think they are cluster wide so only 1 can be there but it can reference various versions of the CRD and activate conversion. So not sure what you are refering to by "instance-id". Can you point to some docs for this?

[phsiao]

Ambassador uses instance-id annotation for example, emissary-ingress/emissary#308. Other operators we worked on has similar concept so we can create virtual partitions of the objects in the same cluster.

For our use cases, we would expect there is a cluster-wide operator and webhook to continue to support existing objects, while testing new operator/webhook by having this virtual partition in some ways. Scoped to a namespace is good, but need also a way to isolate the cluster-wide operator and webhook to process the namespaced objects.

[adriangonz]

That's a really good suggestion @phsiao! I think it would be good to have all these really:

• Support for namespaced (as well as cluster-wide) operators. Webhooks will also need to be scoped to the relevant namespace.
• Support for annotations, similar to how Ambassador uses them. I'm still not sure how these work with webhooks though. Ambassador's operator simply ignores resources with an annotated version different than the operator's. However, webhooks work at the Kubernetes level. I haven't been able to find any example of a configured webhook at the Ambassador project.
• Support for CRD conversion, so that we can have different versions stored at the same time. I'm not sure how kubebuilder handles this at the moment, but I think that the webhook for conversion should also be installed cluster-wide alongside the CRD.

Except the first one, the other two would be out of the scope of this issue.

[ukclivecox]

That's a really good suggestion @phsiao! I think it would be good to have all these really:

* Support for namespaced (as well as cluster-wide) operators. Webhooks will also need to be scoped to the relevant namespace.

* Support for annotations, similar to how Ambassador uses them. I'm still not sure how these work with webhooks though. Ambassador's operator simply ignores resources with an annotated version different than the operator's. However, webhooks work at the Kubernetes level. I haven't been able to find any example of a configured webhook at the Ambassador project.

* Support for CRD conversion, so that we can have different versions stored at the same time. I'm not sure how `kubebuilder` handles this at the moment, but I think that the webhook for conversion should also be installed cluster-wide alongside the CRD.

Except the first one, the other two would be out of the scope of this issue.

I'm looking at solving the first two actually in this issue as they are related. CRD conversion is I agree a separate issue.

[phsiao]

IIRC, we use webhook only on seldon deployment objects, so the annotation will also be there. We just need to be able to deploy multiple mutating webhook configuration, where each knows which virtual partition of objects it needs to work on, and let the others handle the rest.