Metricbeat Beat failed to connect es (xpack enabled)

[paulpuvi06]

Chart version: 7.6.2

Kubernetes version: 1.16

Kubernetes provider: AKS

Helm Version: "v3.1.1"

helm get release output

helm status metricbeat
NAME: metricbeat
LAST DEPLOYED: Thu Apr 30 10:05:32 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1

Describe the bug: Unable to connect to elasticsearch, deployed configmap does not contain username: and password input.

Steps to reproduce:

1. Enable xpack in elasticsearch
2. run - helm install metricbeat elastic/metricbeat --set imageTag=7.6.2 --values metrics.yaml

metrics.yaml

daemonset:
  extraEnvs:
    - name: 'ES_USERNAME'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: username
    - name: 'ES_PASSWORD'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: password
  # Allows you to add any config files in /usr/share/metricbeat
  # such as metricbeat.yml for daemonset
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        metricsets:
          - container
          - node
          - pod
          - system
          - volume
        period: 10s
        host: "${NODE_NAME}"
        hosts: ["${NODE_NAME}:10250"]
        # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        # ssl.verification_mode: "none"
        # If using Red Hat OpenShift remove ssl.verification_mode entry and
        # uncomment these settings:
        #ssl.certificate_authorities:
          #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        processors:
        - add_kubernetes_metadata: ~
      - module: kubernetes
        enabled: true
        metricsets:
          - event
      - module: system
        period: 10s
        metricsets:
          - cpu
          - load
          - memory
          - network
          - process
          - process_summary
        processes: ['.*']
        process.include_top_n:
          by_cpu: 5
          by_memory: 5
      - module: system
        period: 1m
        metricsets:
          - filesystem
          - fsstat
        processors:
        - drop_event.when.regexp:
            system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
      output.elasticsearch:
        username: '${ES_USERNAME}'
        password: '${ES_PASSWORD}'
        hosts: ["elasticsearch-master:9200"]


deployment:
  extraEnvs:
    - name: 'ES_USERNAME'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: username
    - name: ES_PASSWORD'
      valueFrom:
        secretKeyRef:
          name: elastic-credentials
          key: password
  # Allows you to add any config files in /usr/share/metricbeat
  # such as metricbeat.yml for deployment
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        enabled: true
        metricsets:
          - state_node
          - state_deployment
          - state_replicaset
          - state_pod
          - state_container
        period: 10s
        hosts: ["${KUBE_STATE_METRICS_HOSTS}"]
      output.elasticsearch:
        username: '${ES_USERNAME}'
        password: '${ES_PASSWORD}'
        hosts: ["elasticsearch-master:9200"]

metricbeat.yml after deployment does not contain username and password inputs:

 metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["${NODE_NAME}:10255"]
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
    - module: kubernetes
      enabled: true
      metricsets:
        - event
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5
    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

Expected behavior: should connect to es using username and password without an issue.

Provide logs and/or server output (if relevant):

   pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticsearch-master:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Any additional context:

I am able to connect elastic search using same credentials and it works fine with Kibana login as well.

[simonme]

Hi @paulpuvi06!
It seems that the way to configure the metricbeat chart changed on master (see this PR). This will be deployed with versions 7.7.x

I ran into the same issue. What you need to do to fix this is change your values configuration to the old way:

extraEnvs:
  - name: 'ES_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ES_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
metricbeatConfig:
  metricbeat.yml: |
    metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["${NODE_NAME}:10250"]
      # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      # ssl.verification_mode: "none"
      # If using Red Hat OpenShift remove ssl.verification_mode entry and
      # uncomment these settings:
      #ssl.certificate_authorities:
        #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
      processors:
      - add_kubernetes_metadata: ~
    - module: kubernetes
      enabled: true
      metricsets:
        - event
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5
    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
    output.elasticsearch:
      username: '${ES_USERNAME}'
      password: '${ES_PASSWORD}'
      hosts: ["elasticsearch-master:9200"]
  kube-state-metrics-metricbeat.yml: |
    metricbeat.modules:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
      period: 10s
      hosts: ["${KUBE_STATE_METRICS_HOSTS}"]
    output.elasticsearch:
      username: '${ES_USERNAME}'
      password: '${ES_PASSWORD}'
      hosts: ["elasticsearch-master:9200"]

[jmlrt]

FYI 7.7.0 has been released wednesday.