Automatically inject service external IP to the certs SAN

[sebgl]

When users expose the ES cluster through http.service.type, they're hitting a cert validity issue while requesting the cluster:

Failed to connect to backoff(elasticsearch(https://34.66.153.170:9200)): Get https://34.66.153.170:9200: x509: certificate is valid for 10.60.1.15, 127.0.0.1, 10.63.247.8, not 34.66.153.179

It's expected, since the external IP is never injected into ES certs SANs.

The workaround currently is for users to manually inject that IP into the Elasticsearch Spec:

cloud-on-k8s/operators/config/samples/elasticsearch/elasticsearch.yaml

Lines 46 to 51 in 5a8a8a3

	# tls:
	# selfSignedCertificate:
	# subjectAltNames:
	# - ip: 192.168.1.2
	# - ip: 192.168.1.3
	# - dns: elasticsearch-sample.example.com

We should be able to inject that IP automatically, and reconcile on any IP change.
Do we need a way for this to be an opt-in/opt-out feature, or is it OK to just enforce it?

[pebrc]

Why would someone not want this?

[sebgl]

A few things we discussed in today's sync meeting:

• this only matters for ES cluster http endpoint, not individual nodes certs involved in internal node-to-node communication
• we plan to support users providing their own certificate, in such case we don't need to care at all about this. In this issue we only care about the "default" behaviour when people rely on our self-signed certs.
• it may feel wrong to automatically include an external IP to be trusted instead of a DNS name. Most public CAs would not allow that when signing certificates (see https://cabforum.org/baseline-requirements-documents/)
• not having this feature breaks a bit the quickstart experience of exposing an Elasticsearch cluster publicly, if you don't also have a DNS setup
• should we support user-provided services? For example people may want to create their own service to target the Ingest nodes, using a selector on corresponding pods. In such situation, we would also need to inject the IP and service name of these user-provided services.
• we could decide to make this opt-in by having the user provide some logical values in the subjectAltNames section, to also support additional services. Something like:

tls: 
  selfSignedCertificate: 
    subjectAltNames: 
    - ip: 192.168.1.2 
    - dns: elasticsearch-sample.example.com
    - service:
        ref:
            namespace: default
            name: es-ingest-nodes
        serviceName: true
        clusterIP: true
        loadBalancerIP: true

[devdattakulkarni]

I would argue for making this default behavior if the http type is 'LoadBalancer'. This is especially useful for users who will try the Operator on managed Kubernetes services like GKE.

Regarding some of the bullet-points above:

• Signing IPs vs Domain names: I think the LoadBalancer type will typically be used with managed Kubernetes distributions such as GKE, AKS, EKS, etc. As such it seems safe to trust the IP provisioned for the LB.
• User provided Domain name: The Spec can define an optional Domain name field. The expectation here will be user knows how to configure the domain name with the LB IP. From the controller side, if the Domain name is defined, it can use that in cert signing. Otherwise, use LB IP.