Fix handling of unmanaged esRef in Beats Stack Monitoring

[pebrc]

Fixes #6230
Fixes #5880

Beats stack monitoring assumed that the associated Elasticsearch cluster is a local cluster managed by ECK. However a Beats definition as follows would break the stack monitoring feature:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.6.1
  elasticsearchRef:
    secretName: es-ref
  kibanaRef:
    secretName: kibana-ref
  monitoring:
    metrics:
      elasticsearchRefs:
        - secretName: monitoring-metrics-es-ref
    logs:
      elasticsearchRefs:
        - secretName: monitoring-metrics-es-ref

Worse still we would silently swallow the error creating the appearance of normalcy when the situation was in fact not normal.

This PR tries to address these issues by:

• handling the error correctly
• special casing external references to Elasticsearch correctly by
  1. distinguishing between external references and internal references when generating Beats configuration
  2. making a request in case of a remote unmanaged cluster to find out the cluster UUID which we seem to need for stack monitoring configuration (I have not tested what happens if we don't set it )

Known issues:

• if the external Elasticsearch cluster has stack monitoring for the cluster itself turned off then the Beats monitoring data will not show up in the Kibana UI (a known issue that we already document). However monitoring data will still flow into the cluster for use in custom dashboards etc.

[naemono]

So I understand what you're doing here, but I'm a bit unsure as to why we're doing it. We are wanting all communications to unmanaged ES clusters in this code path to be explicitly version validated. We were originally doing this for all requests to unmanaged ES clusters. Was this causing an issue?

[pebrc]

Not sure if I understand your question here. I just renamed this function because the name was misleading. It was called Request originally when in fact it was doing more than just "requesting" but also parsing the result of applying the JSON path expression as a semver version.

I have not changed anything with regards to when we to version validation on associations. All I wanted was a way to do a more generic request through this mechanism so that we can extract the UUID from the remote cluster for the Beats configuration.

[naemono]

Let me try to better explain. We now have both Request, and Version, of which Version is exactly the same code path as Request, and Request now just doesn't parse the version. You also updated all association controllers to use Version, which doesn't change the code path, but makes what's happening a bit clearer. But now in associatedESUUID you call uuid, err := remoteES.Request("/", "{.cluster_uuid}"), which now does not do version validation. I'm wondering why you're not calling .Version here instead, as do all of the other code paths, and if that's the case, do we want to un-export Request?

[pebrc]

I am not doing a version validation because what I am interested in is not the version but the cluster UUID for the Beats config. Version validation is still/already done by the association controller. All I want is to pull a piece of data out of the unmanaged cluster so that we can configure Beats in the same way we do it for ECK managed Elasticsearch clusters.

[naemono]

Okay, so the validation is already being done by the beats controller, which is the piece I was missing here. Thanks again for the explanation

[thbkrkr]

LGTM