Support for multiple namespaces

[skonto]

There is an implementation of a multi-watcher mentioned here: kubernetes/kubernetes#74415 (comment)
We could utilize that. Thoughts?

[skonto]

@liyinan926 what do you think moving to controller-runtime (kubebuilder), it is more uptodate https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/cache/multi_namespace_cache.go
It would be a big change though.

[liyinan926]

It would be great to move to using controller-runtime. But it looks like possible to just use that particular package without doing a huge round of refactoring.

[skonto]

@liyinan926 I understands do you have any insights as what the future might be controller-runtime vs the client lib thing. Long term what should we support?

[liyinan926]

Long term controller-runtime is the way to go I think.

[doctapp]

Any way of deploying multiple spark operators on k8s, like one per namespace?

[zzvara]

I deploy multiple Spark Operators using:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: spark-operator
  namespace: production
spec:
  chart:
    repository: https://googlecloudplatform.github.io/spark-on-k8s-operator
    name: sparkoperator
    version: 1.0.5
  releaseName: spark-operator
  values:
    webhook:
      enabled: true
    resyncInterval: 5
    metrics:
      enable: false
    controllerThreads: 2
    batchScheduler:
      enable: true

and then I have one HelmRelease for each namespace.

[doctapp]

Try removing any one of them and it will corrupt all installations

[zzvara]

Try removing any one of them and it will corrupt all installations

What do you mean by remove and what do you mean by corrupt installation?

[doctapp]

We constantly bring up k8s namespaces and then tear them down, therefore, we install/uninstall many Spark Operators. If you only do the above, then all Spark Operators will become corrupted and no longer work as expected except the most recent version. You need many hacks to make it work correctly.

[zzvara]

@doctapp when you tear down Spark Operator before deleting the namespace and you install it after the namespace is created, what can go wrong?

[doctapp]

CRDs are defined at cluster level along with the web hook, i.e., they are not per-namespace...

…

On Tue, Jan 12, 2021 at 12:46 PM Zoltán Zvara ***@***.***> wrote: @doctapp <https://github.com/doctapp> when you tear down Spark Operator before deleting the namespace and you install it after the namespace is created, what can go wrong? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3OUNBD6GBI4RSNJ4OSNP3SZSDFXANCNFSM4HSGESOA> .

[zzvara]

Is there any way to overwrite CRD's in Helm so that they are not deleted when the chart is removed?

[doctapp]

No standard way other than hacking

…

On Wed, Jan 13, 2021 at 11:17 AM Zoltán Zvara ***@***.***> wrote: Is there any way to overwrite CRD's in Helm so that they are not deleted when the chart is removed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#507 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3OUNBGKU4TJYUL3FFGVWDSZXBQHANCNFSM4HSGESOA> .

[amuraru]

@skonto is the migration to controller-runtime or kubebuilder still pursued here? At some point you started the work in #547 and #658 and was wondering if we should revive that effort.
Thanks!

[skonto]

Hi @amuraru, it has been a while since I last worked on this. Operator multi-tenancy and other concerns is something that comes up often lately. Some frameworks solve part of the problem with multiple-namespace watching but still people are blocked by cluster scoped CRDs for upgrades. Webhooks can also filter requests per namespace so you could have multiple webhooks for example (some operators use that eg. Elastic). Controller runtime is still the way to go but this needs to be discussed with @liyinan926, this is a lot of work but I think there is value there.