Facilitate ECK integration with MutatingAdmissionWebhook

[pebrc]

Integrating with an ECK managed Elasticsearch cluster involves typically:

• getting access to user credentials (username/password)
• getting hold of the necessary TLS certificates

This can be done via two environment variables

          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: my-cluster-es-elastic-user
          - name: ELASTICSEARCH_USER
            value: elastic

Plus a volume/volume mount for the certificates

          volumeMounts:
          - name: es-certs
            mountPath:  /mnt/elastic/tls.crt
            readOnly: true
            subPath: tls.crt
        volumes:
        - name: es-certs
          secret:
            secretName: my-cluster-es-http-certs-public

we could facilitate that kind of integration with a MutatingAdmissionWebhook that is triggered by a label or annotation and is able to inject these snippets into a Pod spec. Something like:
k8s.elastic.co/inject-cluster: my-cluster and optionally to give more control over the injected user: k8s.elastic.co/inject-user: my-custom-user or a set of user roles k8s.elastic.co/inject-roles: kibana_user,monitoring_user ECK would make sure the corresponding user exists and inject its credentials into the Pod spec. It would also make sure the secrets for password and certificates are available in the namespace of the annotated resource.

Concerns:

• copying secrets that give access to Elasticsearch clusters around based on annotations or labels might be undesirable
• complexity of introducing and managing yet another webhook

Benefits:

• fairly generic way of 'integrating' with ECK managed Elasticsearch clusters, without implementing explicit support for every possible integration in ECK itself

[stasrelevanceai]

Hi, we personally right now copy certificate to other namespace where needed with terraform. Works very well on IaC provisioning.