Cannot configure stack monitoring on Beats

[bsakmd]

Bug Report

What did you do?
I set up stack monitoring as per ECK docs for self-monitoring of Elasticsearch cluster and Kibana, and this went well. I can view monitoring data in stack monitoring page. In this case I am populating elasticsearchRefs using the names of Elastic resources since this is the same k8s cluster.

I then wanted to configure stack monitoring in Beats deployed on the same cluster - following the above doc.

What did you expect to see?
Metric data from filebeat in Elasticsearch.

What did you see instead? Under which circumstances?
First of all, there were no changes made to the Beat daemonset at all. When I manually deleted one pod then the newly recreated started with new config.
However this new one went into loop-crash with following error:
Exiting: could not start the HTTP server for the API: listen unix /var/shared/filebeat-eck-filebeat.sock: bind: no such file or directory

When I looked into config generated by ECK it looks like this:

filebeat:
  autodiscover:
    providers:
    - hints:
        default_config:
          paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
          type: container
        enabled: true
      host: ${HOSTNAME}
      type: kubernetes
  modules:
  - log:
      enabled: true
      var:
        paths:
        - /var/log/audit/audit.log
    module: auditd
http:
  enabled: true
  host: unix:///var/shared/filebeat-eck-filebeat.sock
  port: null
logging:
  files:
    path: /usr/share/filebeat/logs
  to_files: true
  to_stderr: false
  to_syslog: false
monitoring:
  enabled: false
output:
  elasticsearch:
    api_key: ${ID}:${API_KEY}
    hosts:
    - https://elasticsearch-es-http.eck.svc:9200
    ssl:
      verification_mode: none
processors:
- add_cloud_metadata: null
- add_host_metadata: null
setup:
  dashboards:
    enabled: true
  kibana:
    host: https://kibana-kb-http.eck.svc:5601
    ssl:
      certificate_authorities:
      - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
      verification_mode: none

When I manually tweaked the path it started successfully:
host: unix:///var/run/filebeat-eck-filebeat.sock

I tried to put elasticsearch reference via secret instead of instance reference but it made no difference.

I have upgrade the ECK from v2.5.0 to v2.6.2 but it gives same results.

Environment

• ECK version:

  2.5.0 & 2.6.2

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • Cloud: Azure, self-manged VMs
  • Kubernetes distribution: Openshift 4.11.18

  for each of them please give us the version you are using

$ oc version
bash-4.4 ~ $ oc version
Client Version: 4.11.2
Kustomize Version: v4.5.4
Server Version: 4.11.18
Kubernetes Version: v1.24.6+5658434

• Resource definition:
  https://gist.github.com/bsakmd/82f1cd083fc207dcad0fe0769309dd65

• Logs:

`Exiting: could not start the HTTP server for the API: listen unix /var/shared/filebeat-eck-filebeat.sock: bind: no such file or directory`

[barkbay]

Are you running Windows or Linux k8s nodes?

If windows you may actually hit this limitation:

func GetStackMonitoringSocketURL(beat *v1beta1.Beat) string {
	// TODO: Enable when Beats as containers in Windows is supported: https://github.com/elastic/beats/issues/16814
	// if runtime.GOOS == "windows" {
	// 	return fmt.Sprintf("npipe:///%s-%s-%s.sock", beat.Spec.Type, beat.GetNamespace(), beat.GetName())
	// }
	return fmt.Sprintf("unix:///var/shared/%s-%s-%s.sock", beat.Spec.Type, beat.GetNamespace(), beat.GetName())
}

[bsakmd]

Are you running Windows or Linux k8s nodes?

If windows you may actually hit this limitation:

func GetStackMonitoringSocketURL(beat *v1beta1.Beat) string {
	// TODO: Enable when Beats as containers in Windows is supported: https://github.com/elastic/beats/issues/16814
	// if runtime.GOOS == "windows" {
	// 	return fmt.Sprintf("npipe:///%s-%s-%s.sock", beat.Spec.Type, beat.GetNamespace(), beat.GetName())
	// }
	return fmt.Sprintf("unix:///var/shared/%s-%s-%s.sock", beat.Spec.Type, beat.GetNamespace(), beat.GetName())
}

we are running Linux k8s nodes

[barkbay]

I did a quick test to check if there was any regression. I managed to deployed a stack monitored instance including Elasticsearch, Kibana and Beats on OpenShift 4.11.21 using ECK 2.6.2 (deployed using the OperatorHub):

In the Beats logs I can see the following:

{
	"log.level": "info",
	"@timestamp": "2023-01-27T08:35:22.008Z",
	"log.logger": "api",
	"log.origin": {
		"file.name": "api/server.go",
		"file.line": 65
	},
	"message": "Metrics endpoint listening on: /var/shared/filebeat-monitored-filebeat.sock (configured: unix:///var/shared/filebeat-monitored-filebeat.sock)",
	"service.name": "filebeat",
	"ecs.version": "1.6.0"
}

The manifests I used are here.

Exiting: could not start the HTTP server for the API: listen unix /var/shared/filebeat-eck-filebeat.sock: bind: no such file or directory

/var/shared is supposed to be an empty dir mounted in the Beat Pods:

    - mountPath: /var/shared
      name: shared-data

See also the source code here:

	// Add shared volume for Unix socket between containers.
	sidecar.Container.VolumeMounts = append(sidecar.Container.VolumeMounts, corev1.VolumeMount{
		Name:      "shared-data",
		MountPath: "/var/shared",
		ReadOnly:  false,
	})

Could you check if there is any error in the operator logs?

[bsakmd]

Hi,
Thanks for investigating this! I have tested your setup and it works.
It seems that the main difference with mine is that I don't use elasticsearchRef for beat output.
Instead I put explicitly ES output parameters - I use that for sake of API keys authentication.

When I switched to using elasticsearchRef/kibanaRef the pod started properly, there is side container in the pod and I see metrics in Kibana > Stack Monitoring.

I think this stack monitoring should work even without using references.

[naemono]

Closing, as I believe this was resolved in #6482