[Agent] Dynamic inputs fail to get validated

[ChrsMark]

It was found during #23679. Dynamic inputs setup to automatically discover scheduler and controllermanager Pods does not work right now and we get the following error:

2021-01-25T15:36:29.224Z	DEBUG	application/periodic.go:40	Failed to read configuration, error: could not emit configuration: could not create the AST from the configuration: missing field accessing 'inputs' (source:'/etc/agent.yml')

Agent version:

Daemon: 7.11.0-SNAPSHOT (build: 5691b7c654663ee51f1d347ce02bda332fb300e7 at 2021-01-25 09:48:16 +0000 UTC)
Binary: 7.11.0-SNAPSHOT (build: 5691b7c654663ee51f1d347ce02bda332fb300e7 at 2021-01-25 09:48:16 +0000 UTC)

agent.yml:

    id: ef9cc740-5bf0-11eb-8b51-39775155c3f5
    revision: 2
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
    agent:
      monitoring:
        enabled: true
        use_output: default
        logs: true
        metrics: true
    providers.kubernetes:
      node: ${NODE_NAME}
      scope: node
    inputs:
      - id: 934ef8aa-ed19-405b-8160-ebf62e3d32f8
        name: kubernetes-node-metrics
        revision: 1
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 0.2.8
        data_stream:
          namespace: default
        streams:
          - id: >-
              kubernetes/metrics-kubernetes.controllermanager-3d50c483-2327-40e7-b3e5-d877d4763fe1
            data_stream:
              dataset: kubernetes.controllermanager
              type: metrics
            metricsets:
              - controllermanager
            hosts:
              - '${kubernetes.pod.ip}:10252'
            period: 10s
            #condition: startsWith(${kubernetes.pod.name}, "kube-controller-manager")
            condition: ${kubernetes.pod.labels.component} == 'kube-controller-manager'
          - id: >-
              kubernetes/metrics-kubernetes.scheduler-3d50c483-2327-40e7-b3e5-d877d4763fe1
            data_stream:
              dataset: kubernetes.scheduler
              type: metrics
            metricsets:
              - scheduler
            hosts:
              - '${kubernetes.pod.ip}:10251'
            period: 10s
            #condition: startsWith(${kubernetes.pod.name}, "kube-scheduler")
            condition: ${kubernetes.pod.labels.component} == 'kube-scheduler'
          - id: >-
              kubernetes/metrics-kubernetes.proxy-3d50c483-2327-40e7-b3e5-d877d4763fe1
            data_stream:
              dataset: kubernetes.proxy
              type: metrics
            metricsets:
              - proxy
            hosts:
              - 'localhost:10249'
            period: 10s
          - id: >-
              kubernetes/metrics-kubernetes.container-934ef8aa-ed19-405b-8160-ebf62e3d32f8
            data_stream:
              dataset: kubernetes.container
              type: metrics
            metricsets:
              - container
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none

Converting ${NODE_NAME} placeholders to ${env.NODE_NAME} does not fix the problem and even if we remove all other datastream configs and leave only the dynamic one it still gives the error.

In addition, if we remove the dynamic inputs part and have ${env.NODE_NAME} we still get the same error.

Also a condition like condition: startsWith(${kubernetes.pod.name}, "kube-controller-manager") fails and the inspect command gives the following:

condition "startsWith(${kubernetes.pod.name}, "kube-controller-manager")" evaluation failed: startsWith: accepts exactly 2 string arguments; recieved *eql.null and string

Related to #23613

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

I think there are two problems here:

1. I think referencing environment variables in the Elastic Agent is not working, I think this an Agent concerns.

      node: ${NODE_NAME}
      scope: node

2. eql.Null I think it's either a side effect of 1, or a problem in the K8s provider.

@blakerouse are my theories correct? I think 1 should be done by the Agent team and 2 (if its not a side effect of 1) should be done by integrations?

[blakerouse]

I believe the startsWith function doesn't handle a null value being passed into the string comparison. It should handle that and not raise an error.

I think both issues are from Agent side, I need to investigate this more. I think there is something else core going wrong.