Add etcd metrics, Prometheus scrapes, and Grafana dash

[dghubble]

• Use etcd v3.3 --listen-metrics-urls to expose only metrics data via http://0.0.0.0:2381 on controllers
• Add Prometheus discovery for etcd peers on controller nodes
• Enables etcd related alerts and populates the etcd Grafana dashboard
• Note: These benefits require the optional Prometheus and Grafana addons be applied.

Hold off on allowing workers firewall access (can't think of any concrete concern with workloads seeing this). Move Prometheus to a controller node for a while (maybe drop). Adjust firewall rules now that Prometheus can run on a controller, rather than a worker

Made possible by:

• Expose /metrics an a separate port etcd-io/etcd#8060 thanks @gyuho @brancz
• *: support additional '/metrics' endpoints etcd-io/etcd#8242

Closes #114

[dghubble]

The HighNumberOfFailedGRPCRequests Prometheus rule triggers noisy alerts for etcdserverpb.Watch with grpc_code Unavailable. The rule gets triggered across platforms, even with very fast SSDs and I can manually use etcdctl to watch keys without issue. That rule probably hasn't been exercised as much since few distros monitor etcd yet.

#176 was to address an alert that fired specifically on AWS because of slow disks.

[brancz]

We've seen that a couple times before, and I believe that it's because of etcd-io/etcd#9166 for which doesn't seem like the "fix" landed in 3.3. I believe even then it will require a change to the alerting rules to ignore cancelled connections.

[dghubble]

For now I think I'll drop the two noisy alerts. Even without them, this change adds etcd alerts which weren't active before and populates the one page in Grafana that was empty before.

Its a good feedback loop too! - using the new etcd alerts motivated switching to faster disks on AWS for the v1.10 release (other platforms were fast enough).