Support custom Secret for associating external Elastic resources not managed by ECK

[thbkrkr]

This adds support for using a custom secret to associate an ECK-managed Elastic resource with an external non-ECK-managed Elastic resource.

Resolves #5078.

Changes

Add a field secretName: string to the common.ObjectSelector struct used to reference a resource in an association. This change impacts the API of all the Elastic resources.

  elasticsearchRef:
    secretName: test-ext-es-ref

Use an ObjectSelector instead of a NamedNamespace for the type of the key of the map of AssociationConf (used in assocations that support multiple references) to take in account the new field.

Change the signature of ReferencedResourceVersion: func(c k8s.Client, esRef commonv1.ObjectSelector) and implement for all types of Elastic resources how to get the version of the referenced external resource (requesting its HTTP API and extracting the version from the response using a jsonPath).

Basically the main change consists to shortcut the reconciliation loop of the association controller, if a referenced object is a of type Secret. Then, get the version of the referenced external resource, which is the opportunity to check that we can reach it and create an association config (which stored in an annotation of the associated resource) with the auth and CA Secrets pointing to the custom Secret..

    "association.k8s.elastic.co/es-conf-620140190": "{
	\"authSecretName\":\"test-ext-es-ref\",\"authSecretKey\":\"password\",
	\"caCertProvided\":false,\"caSecretName\":\"\",
	\"url\":\"https://test.es.abc-0.xyz.com:1234\",
	\"version\":\"7.16.3\"}",

In the Stack Monitoring association 'es-es', we configured MetricBeat with a predefined user that exists when the referenced ES is managed by ECK. This is changed to use the username and password defined in the custom secret instead.

6ad15eb has been superseeded by #5277.

Notes

Before FleetServer 8.0, there is no API to get the version, so we just set unknown for the version and the compatilibity check is skipped.

The user corresponding to the username, password referenced in the custom secret must have the appropriate privileges depending on the type of association. When the referenced resource is managed by ECK, a user with a specific role with the right privileges is created specifically for the association.

Currently, there is no way to get the FleetServer version but it should be available in a next release. This means we need a fallback for now and for older versions. It could be:

• do not support custom secret for the Agent-FleetServer association
• use the associated version, which is equivalent to ignore the compatibility check
• expect a version in the secret

Remains to be done

• Do something for the

Test

Create an Elasticsearch cluster test-ext-es somewhere and then a secret with its URL, username, password and optionally a CA.

 echo 'apiVersion: v1
kind: Secret
metadata:
  name: test-ext-es-ref
stringData:
  url: https://test.es.abc-0.xyz.com:1234
  username: hey
  password: 1234567890' | kubectl apply -f-

Let's use this ES as a monitoring cluster by creating a Kibana m associated to an Elasticsearch m with the monitoring configured to send metrics and logs to test-ext-es. And create a Kibana gw associated to test-es-ref to display the Stack Monitoring page.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: m
spec:
  version: 8.0.0
  monitoring:
    metrics:
      elasticsearchRefs:
      - secretName: test-ext-es-ref
    logs:
      elasticsearchRefs:
      - secretName: test-ext-es-ref
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: m
spec:
  version: 8.0.0
  monitoring:
    logs:
      elasticsearchRefs:
      - secretName: test-ext-es-ref
    metrics:
      elasticsearchRefs:
      - secretName: test-ext-es-ref
  count: 1
  config:
    xpack.security.audit.enabled: true
  elasticsearchRef:
    name: t
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: gw
spec:
  version: 7.16.3
  count: 1
  elasticsearchRef:
    secretName: test-ext-es-ref

[thbkrkr]

Jenkins test this please

[pebrc]

Nice work! I tested a monitoring setup sending data to Elastic Cloud and a setup with an external FleetServer. The monitoring setup worked right away (even though the data did not show up in Stack monitoring but that might have reasons in Elasticsearch or our stack monitoring setup in general and is not necessarily related to this PR, and to be clear data was ingested)

Fleet Server was a different story and I think more work needs to be done to enable this. For example I had of course to hardcode the version of the external Fleet Server but also in the agent controller itself are more changes needed:

diff --git a/pkg/controller/agent/pod.go b/pkg/controller/agent/pod.go
index 434783d6a..e378f0c48 100644
--- a/pkg/controller/agent/pod.go
+++ b/pkg/controller/agent/pod.go
@@ -6,6 +6,7 @@ package agent
 
 import (
        "fmt"
+       pkgerrors "github.com/pkg/errors"
        "hash"
        "path"
        "sort"
@@ -175,22 +176,24 @@ func buildPodTemplate(params Params, fleetCerts *certificates.CertificatesSecret
 }
 
 func amendBuilderForFleetMode(params Params, fleetCerts *certificates.CertificatesSecret, builder *defaults.PodTemplateBuilder, configHash hash.Hash) (*defaults.PodTemplateBuilder, error) {
-       esAssociation, err := getRelatedEsAssoc(params)
-       if err != nil {
-               return nil, err
-       }
+       if params.Agent.Spec.FleetServerRef.Type != commonv1.ObjectTypeSecret {
+               esAssociation, err := getRelatedEsAssoc(params)
+               if err != nil {
+                       return nil, err
+               }
 
-       builder, err = applyRelatedEsAssoc(params.Agent, esAssociation, builder)
-       if err != nil {
-               return nil, err
-       }
+               builder, err = applyRelatedEsAssoc(params.Agent, esAssociation, builder)
+               if err != nil {
+                       return nil, err
+               }
 
-       err = writeEsAssocToConfigHash(params, esAssociation, configHash)
-       if err != nil {
-               return nil, err
+               err = writeEsAssocToConfigHash(params, esAssociation, configHash)
+               if err != nil {
+                       return nil, err
+               }
        }
 
-       builder, err = applyEnvVars(params, builder)
+       builder, err := applyEnvVars(params, builder)
        if err != nil {
                return nil, err
        }
@@ -381,7 +384,7 @@ func getAssociatedFleetServer(params Params) (commonv1.Associated, error) {
                reconcile.Request{NamespacedName: fsRef.NamespacedName()},
                &fs,
        ); err != nil {
-               return nil, err
+               return nil, pkgerrors.Wrap(err, "while fetching associated fleet server")
        }
 
        return &fs, nil
@@ -497,9 +500,11 @@ func getFleetSetupFleetEnvVars(agent agentv1alpha1.Agent, client k8s.Client) (ma
                }
 
                if assoc != nil {
-                       fleetCfg[FleetCA] = path.Join(certificatesDir(assoc), CAFileName)
                        fleetCfg[FleetURL] = assoc.AssociationConf().GetURL()
                }
+               if assoc != nil && assoc.AssociationConf().CACertProvided {
+                       fleetCfg[FleetCA] = path.Join(certificatesDir(assoc), CAFileName)
+               }
        }

This lead me to question the usefulness of this feature outside of our stack monitoring feature where users have no other good way of configuring an alternative connection.

Especially for FleetServer I think users are better off configuring the connection manually via the podTemplate than via this new mechanism:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata: 
  name: elastic-agent
spec:
  version: 7.16.3
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        containers:
          - name: agent
            env:
              - name: FLEET_ENROLL
                value: "1"
              - name: FLEET_ENROLLMENT_TOKEN
                value: a-valid-token==
              - name: FLEET_URL
                value: https://my-external-fleet-server.es.io

If they want to use the association mechanism users have to configure two secrets one for Kibana and one for FleetServer. From a security standpoint the solution with the enrolment token is preferable over the wide ranging privileges required to generate an enrolment token via Kibana as we do it in ECK.

I guess the only benefit over manual configuration I can see in cases like Kibana->ES is that we auto-rotate the associated Pods once the content of the external association secret changes which can be useful if the users users not universally trusted certificates and needs to configure a CA cert. But maybe I am missing some other benefit?

Given that we have shared association implementation I think I am OK with keeping this feature also for associations where it is less useful, though.

[pebrc]

I think some of the changes sketched out in #5240 (review) still need to be implemented to make this work with Fleet/FleetServer as well.

Stack monitoring works great already! We probably want a follow up task to support CloudID.

Documentation could be added already in this PR.

[thbkrkr]

Yes, I will add the documentation and we can add the CloudID support in a follow-up.

Apologies for FleetServer, I naively thought that it will be fixed by #5314 but it's more complicated than that because we can have an unmanaged transitive Elasticsearch. We depend on transitive ES in two places: to create the role/user in the association controller and also to register its CA in the container Agent trust store.
I am testing these different combinations:

• local Agent -> local Kibana -> external Elasticsearch
• local Agent -> external Kibana
• local Agent -> local FleetServer -> external Elasticsearch
• local Agent -> external FleetServer

I'm gonna push my latest wip version but I need to do some more testing.

[pebrc]

LGTM 🎉

[barkbay]

I left a few comments, still need to do some tests.

[barkbay]

Unless I'm missing something the remote cluster logic is not able to handle the secretName field.

[thbkrkr]

Oh, I missed that. I proposed to add a LocalObjectSelector: bad6edc.

[barkbay]

Maybe then include LocalObjectSelector in ObjectSelector to avoid code duplication and suggest the "inheritance":

// ObjectSelector defines a reference to a Kubernetes object which can be an Elastic resource managed by the operator
// or a Secret describing an external Elastic resource not managed by the operator.
type ObjectSelector struct {
	LocalObjectSelector

	// SecretName is the name of an existing Kubernetes secret that contains connection information for associating an
	// Elastic resource not managed by the operator. The referenced secret must contain the following:
	// - `url`: the URL to reach the Elastic resource
	// - `username`: the username of the user to be authenticated to the Elastic resource
	// - `password`: the password of the user to be authenticated to the Elastic resource
	// - `ca.crt`: the CA certificate in PEM format (optional).
	// This field cannot be used in combination with the other fields name, namespace or serviceName.
	SecretName string `json:"secretName,omitempty"`
}

// WithDefaultNamespace adds a default namespace to a given ObjectSelector if none is set.
func (o ObjectSelector) WithDefaultNamespace(defaultNamespace string) ObjectSelector {
	if len(o.Namespace) > 0 {
		return o
	}
	return ObjectSelector{
		LocalObjectSelector: o.LocalObjectSelector.WithDefaultNamespace(defaultNamespace),
		SecretName:          o.SecretName,
	}
}

[thbkrkr]

One issue with the inheritance is that it makes the syntax heavier:

commonv1.ObjectSelector{
  LocalObjectSelector: commonv1.LocalObjectSelector{
    Name:      "elasticsearch-sample",
    Namespace: "e2e-mercury",
  },
},

Considering the amount of shared code, I find it's not worth it here.

[barkbay]

LGTM

[thbkrkr]

run/e2e-tests match=TestExternalESStackMonitoring

=> https://devops-ci.elastic.co/job/cloud-on-k8s-pr-e2e-tests/2 ✔️