Add anonymous etcd metrics endpoints so that prometheus can scrape etcd metrics without requiring root permissions

[ericsuhong]

Right now, etcd /metrics endpoint is exposed via cert-protected endpoint: 127.0.0.1:2379.

This is requiring prometheus to run as a root to access /etc/kubernetes/certs/etcd* certificartes in order to talk with etcd/metrics endpoint.

etcd added a support to run /metrics endpoint in a separate port anonymously, by simply adding following arguments at the startup (etcd-io/etcd#8060):

--listen-metrics-urls=http://[masterIP]:9379

(9379 is a default port allocated by prometheus for etcd exporters: https://github.com/prometheus/prometheus/wiki/Default-port-allocations).

This will allow prometheus to scrape etcd metrics anonymously without requiring a root access.

I verified that manually adding this argument allowed prometheus to scrape metrics anonymously.

This also doesn't break backward compatibility as it keeps old metrics endpoint as well.

Explain why AKS Engine needs it

This will allow prometheus to scrape etcd metrics without requiring root access nor access to certificates.

Describe the solution you'd like

Add --listen-metrics-urls=http://[masterIP]:9379 as an argument at etcd startup.

Describe alternatives you've considered

Run prometheus-server in master nodes as a root, with /etc/kubernetes/certs folder mounted to grab etcd CA/client certs.

securityContext:
      runAsNonRoot: false
      runAsUser: 0

Then, I used following scrape rule to scrape etcd metrics:

- job_name: etcd
  scheme: https
  tls_config:
    ca_file: /etc/kubernetes/certs/ca.crt
    cert_file: /etc/kubernetes/certs/etcdclient.crt
    key_file: /etc/kubernetes/certs/etcdclient.key
  static_configs:
    - targets: [ 10.240.0.4:2379, 10.240.0.5:2379, ... ]

This requires prometheus-server to run as root, which is BAD!

Additional context

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DavidParks8]

@jackfrancis it looks like this was closed by automation. Can you please keep it open until it is implemented?

[idanlevin]

+1 we really need this one.. @jackfrancis @ritazh

[ritazh]

Thanks for opening the issue and the proposed solution! PRs are welcome!

[idanlevin]

Thanks for opening the issue and the proposed solution! PRs are welcome!

Thanks @ritazh ! We will contribute for sure 👍