Disable connect-injector Dynamic Admission Control in kube-system namespace

[davidesalerno]

When a Kubernetes cluster-wide crash happens and all the services in all the namespaces (also the system ones like kube-system) will crash, if in the exclusion list of the Consul Connect Injector Mutating Webhook is not included also the systems namespace (like kube-systems), when all the system services will be restarted after the crash they will try to reach the Mutating Webhook services and it is still down (due to the unavailability of the services in the kube-system namespace for example), all the Kubernetes cluster won't restart at all.

It will be really useful the chance to add the kube-system and other critical namespaces in the NotIn condition to avoid this issue that could be solved only manually deleting the mutatingwebhook configuration and re-applying it when Kubernetes is restarted.

consul-k8s/charts/consul/templates/connect-inject-mutatingwebhook.yaml

Line 20 in 24be51c

values: [ {{ template "consul.name" . }} ]

[davidesalerno]

Reading the documentation the way to do this is using a namepaceSelector -> https://www.consul.io/docs/k8s/helm#v-connectinject-namespaceselector

[lkysow]

#726 will make this the default too so we don't require users to set it.

@davidesalerno it's worth nothing that that PR only works for kube 1.21+ though since they only set that label there by default on that version

[davidesalerno]

@lkysow So on kube < 1.21.1 to achieve the same result, should we put the kube-system in the k8sdenynamespaces (https://www.consul.io/docs/k8s/helm#v-connectinject-k8sdenynamespaces) ?

Or maybe we could simply add a label to the kube-system namespace.

[lkysow]

You would need to label it. The k8sdenynamespaces is only respected by the injector so k8s will still send pod mutation requests for kube-system to the injector. The namespaceSelector is the only way to get kube to not send those mutation requests.

[trenslow]

@lkysow Which version of Consul should we expect to have this change of default?

[lkysow]

Hi Tyler, this is still in review so should make it into our next release. You can get the feature now though by setting:

connectInject:
  namespaceSelector: |
    matchExpressions:
      - key: "kubernetes.io/metadata.name"
        operator: "NotIn"
        values: ["kube-system","local-path-storage"]