Make use of secure port when accessing Kubelet API

[ChrsMark]

What does this PR do?

This PR switches Metricbeat k8s manifests and docs to point to Kubelet secure port over https instead of the insecure port.

Why is it important?

Insecure port of Kubelet (10255/TCP) is now less common and discouraged and also in most cases it is not enabled by default (requiring to restart kubelet with --read-only-port flag)

Related to elastic/beats#16063

[jmlrt]

jenkins test this please

[ChrsMark]

Hey @jmlrt . What is the status on this? Is this valid or is anything else missing here?

[jmlrt]

jenkins test this please

[jmlrt]

Hi @ChrsMark,
I'll take a look at your PR today.
Meanwhile tests are failing, can you merge master on your branch?

[ChrsMark]

Updated with the latest master. Not sure if the latest failure is related though.

[jmlrt]

@ChrsMark This is really strange, we have a test which queries elasticsearch for q=metricset.name:container%20AND%20kubernetes.container.name:metricbeat and verify that it match the metricbeat index here.

This test is failing in your PR because there is no document matching metricset.name:container and kubernetes.container.name:metricbeat. In addition of our failing test on GKE, I could also reproduce it locally on Docker for Mac.

In the same time, The test on master branch is still working well (see logs here).

[ChrsMark]

Hi @jmlrt ! Could I somehow check what is the output of Metricbeat pod? If there is a way to reproduce it locally feel free to mention and I could give it a shot.

To share some content here, with this change Metricbeat will try to query for metrics from Kubelet's API secure port instead of the insecure which was the previous one in the configuration. So I'm wondering if for some reason this port is not accessible in the testing env maybe because Kubelet is configured without this port enabled or for some reason this port is not exposed.

[jmlrt]

Hi Chris,

Here are the logs of a metricbeat pod when deploying your metricbeat chart from this PR on a GKE 1.15 cluster with default config:

2020-04-03T10:38:59.361Z        INFO    instance/beat.go:622    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2020-04-03T10:38:59.361Z        INFO    instance/beat.go:630    Beat ID: fb4eeb21-eef7-49ae-9878-0fb6643c76d3
2020-04-03T10:38:59.396Z        INFO    [api]   api/server.go:62        Starting stats endpoint
2020-04-03T10:38:59.397Z        INFO    [api]   api/server.go:64        Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2020-04-03T10:38:59.397Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "fb4eeb21-eef7-49ae-9878-0fb6643c76d3"}}}
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:27:27.000Z", "version": "7.6.2"}}}
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.8"}}}
2020-04-03T10:38:59.398Z        INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-02T12:49:11Z","containerized":true,"name":"metricbeat-metricbeat-6nlxd","ip":["127.0.0.1/8","10.4.3.7/24"],"kernel_version":"4.19.104+","mac":["46:ba:fe:e1:f9:e9"],"os":{"family":"redhat","
platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-04-03T10:38:59.399Z        INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fs
etid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid"
,"setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-04-03T10:38:58.630Z"}}}
2020-04-03T10:38:59.399Z        INFO    instance/beat.go:298    Setup Beat: metricbeat; Version: 7.6.2
2020-04-03T10:38:59.399Z        INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.6.2' as ILM is enabled.
2020-04-03T10:38:59.399Z        INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticsearch-master:9200
2020-04-03T10:38:59.399Z        INFO    [publisher]     pipeline/module.go:110  Beat name: metricbeat-metricbeat-6nlxd
2020-04-03T10:38:59.426Z        INFO    add_kubernetes_metadata/kubernetes.go:70        add_kubernetes_metadata: kubernetes env detected, with version: v1.15.11-gke.1
2020-04-03T10:38:59.426Z        INFO    kubernetes/util.go:94   kubernetes: Using pod name metricbeat-metricbeat-6nlxd and namespace default to discover kubernetes node
2020-04-03T10:38:59.438Z        INFO    kubernetes/util.go:100  kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 discovered by in cluster pod node query
2020-04-03T10:38:59.539Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.541Z        INFO    kubernetes/util.go:79   kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 provided in the config
2020-04-03T10:38:59.541Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.542Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.543Z        INFO    kubernetes/util.go:79   kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 provided in the config

It seems that secure ports isn't available from inside the pods with default GKE config.

[ChrsMark]

After discussing and debugging it with @jmlrt, it was found that the problem was a missing hostNetwork: true from the pod which is present in Beat's repository manifests (https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml#L112) and make it possible to access Kubelet API from within the pod using HOSTNAME.

Since it is not possible to add it in helm charts too (see #391 (comment) for more info), we switch from using HOSTNAME to NODE_NAME in order to access the Kubelet API.

For reference if hostNetwork is not set, then HOSTNAME has as value the name of pod, and this why sth like curl -H "Authorization: Bearer $token" https://${HOSTNAME}:10250/stats/summary --insecure is not gonna work.

@exekias any comments on this, since we have it with HOSTNAME in Beat's repo?

[exekias]

This sounds reasonable. Also since we are now exposing NODE_NAME we can switch to using it too to avoid confusion, WDYT?

[jmlrt]

⛴ Thanks for this PR 👍