Add kube-state-metrics based metricsets (#4253)

Add k8s metrics extracted from kube-state-metrics services. This adds info specially useful about:

* Status of deployments, pods and replicasets
* Resource limits, requests and usage by pods
* Node available resources
* I have updated existing dashboard with this kind of info

CHANGELOG.asciidoc

@@ -59,6 +59,7 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha1...master[Check the HEAD d
 
 - Add macOS implementation of the system diskio metricset. {issue}4144[4144]
 - Add process_summary metricset that records high level metrics about processes. {pull}4231[4231]
+- Add `kube-state-metrics` based metrics to `kubernetes` module {pull}4253[4253]
 
 *Packetbeat*
 

NOTICE

@@ -1181,6 +1181,73 @@ github.com/ericchiang/k8s
 Apache License
 
 
+--------------------------------------------------------------------
+github.com/golang/protobuf
+--------------------------------------------------------------------
+Go support for Protocol Buffers - Google's data interchange format
+
+Copyright 2010 The Go Authors.  All rights reserved.
+https://github.com/golang/protobuf
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+    * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+--------------------------------------------------------------------
+github.com/matttproud/golang_protobuf_extensions
+--------------------------------------------------------------------
+Apache License
+
+-------NOTICE-----
+Copyright 2012 Matt T. Proud (matt.proud@gmail.com)
+
+--------------------------------------------------------------------
+github.com/prometheus/client_model
+--------------------------------------------------------------------
+Apache License
+
+-------NOTICE-----
+Data model artifacts for Prometheus.
+Copyright 2012-2015 The Prometheus Authors
+
+This product includes software developed at
+SoundCloud Ltd. (http://soundcloud.com/).
+
+--------------------------------------------------------------------
+github.com/prometheus/common
+--------------------------------------------------------------------
+Apache License
+
+-------NOTICE-----
+Common libraries shared by Prometheus Go components.
+Copyright 2015 The Prometheus Authors
+
+This product includes software developed at
+SoundCloud Ltd. (http://soundcloud.com/).
+
 --------------------------------------------------------------------
 github.com/Microsoft/go-winio
 --------------------------------------------------------------------
@@ -2138,38 +2205,6 @@ The above copyright notice and this permission notice shall be included in all c
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
---------------------------------------------------------------------
-github.com/matttproud/golang_protobuf_extensions
---------------------------------------------------------------------
-Apache License
-
--------NOTICE-----
-Copyright 2012 Matt T. Proud (matt.proud@gmail.com)
-
---------------------------------------------------------------------
-github.com/prometheus/client_model
---------------------------------------------------------------------
-Apache License
-
--------NOTICE-----
-Data model artifacts for Prometheus.
-Copyright 2012-2015 The Prometheus Authors
-
-This product includes software developed at
-SoundCloud Ltd. (http://soundcloud.com/).
-
---------------------------------------------------------------------
-github.com/prometheus/common
---------------------------------------------------------------------
-Apache License
-
--------NOTICE-----
-Common libraries shared by Prometheus Go components.
-Copyright 2015 The Prometheus Authors
-
-This product includes software developed at
-SoundCloud Ltd. (http://soundcloud.com/).
-
 --------------------------------------------------------------------
 github.com/google/uuid
 --------------------------------------------------------------------

filebeat/docs/fields.asciidoc

@@ -684,12 +684,14 @@ The logged message.
 
 
 [[exported-fields-kubernetes-processor]]
-== Kubernetes info Fields
+== kubernetes Fields
 
+beta[]
 Kubernetes metadata added by the kubernetes processor
 
 
 
+
 [float]
 === kubernetes.pod.name
 
@@ -730,6 +732,14 @@ type: keyword
 Kubernetes container name
 
 
+[float]
+=== kubernetes.container.image
+
+type: keyword
+
+Kubernetes container image
+
+
 [[exported-fields-log]]
 == Log File Content Fields
 

heartbeat/docs/fields.asciidoc

@@ -411,12 +411,14 @@ type: long
 Duration in microseconds
 
 [[exported-fields-kubernetes-processor]]
-== Kubernetes info Fields
+== kubernetes Fields
 
+beta[]
 Kubernetes metadata added by the kubernetes processor
 
 
 
+
 [float]
 === kubernetes.pod.name
 
@@ -457,6 +459,14 @@ type: keyword
 Kubernetes container name
 
 
+[float]
+=== kubernetes.container.image
+
+type: keyword
+
+Kubernetes container image
+
+
 [[exported-fields-resolve]]
 == Host Lookup Fields
 

libbeat/processors/kubernetes/_meta/fields.yml

@@ -1,30 +1,41 @@
 - key: kubernetes
-  title: Kubernetes info
+  title: kubernetes
   description: >
+    beta[]
+
     Kubernetes metadata added by the kubernetes processor
+  short_config: false
   anchor: kubernetes-processor
   fields:
-    - name: kubernetes.pod.name
-      type: keyword
-      description: >
-        Kubernetes pod name
+    - name: kubernetes
+      type: group
+      fields:
+        - name: pod.name
+          type: keyword
+          description: >
+            Kubernetes pod name
+
+        - name: namespace
+          type: keyword
+          description: >
+            Kubernetes namespace
 
-    - name: kubernetes.namespace
-      type: keyword
-      description: >
-        Kubernetes namespace
+        - name: labels
+          type: object
+          description: >
+            Kubernetes labels map
 
-    - name: kubernetes.labels
-      type: object
-      description: >
-        Kubernetes labels map
+        - name: annotations
+          type: object
+          description: >
+            Kubernetes annotations map
 
-    - name: kubernetes.annotations
-      type: object
-      description: >
-        Kubernetes annotations map
+        - name: container.name
+          type: keyword
+          description: >
+            Kubernetes container name
 
-    - name: kubernetes.container.name
-      type: keyword
-      description: >
-        Kubernetes container name
+        - name: container.image
+          type: keyword
+          description: >
+            Kubernetes container image

libbeat/tests/system/beat/beat.py

@@ -426,7 +426,9 @@ def extract_fields(doc_list, name):
 
         # TODO: Make fields_doc path more generic to work with beat-generator
         with open(fields_doc, "r") as f:
-            path = os.path.abspath(os.path.dirname(__file__) + "../../../../_meta/fields.common.yml")
+            path = os.path.abspath(os.path.dirname(__file__) + "../../../../_meta/fields.generated.yml")
+            if not os.path.isfile(path):
+                path = os.path.abspath(os.path.dirname(__file__) + "../../../../_meta/fields.common.yml")
             with open(path) as f2:
                 content = f2.read()
 

metricbeat/docker-compose.yml

@@ -26,6 +26,7 @@ services:
       - ${PWD}/module/jolokia/_meta/env
       - ${PWD}/module/kafka/_meta/env
       - ${PWD}/module/kibana/_meta/env
+      - ${PWD}/module/kubernetes/_meta/env
       - ${PWD}/module/memcached/_meta/env
       - ${PWD}/module/mongodb/_meta/env
       - ${PWD}/module/mysql/_meta/env
@@ -51,6 +52,8 @@ services:
       jolokia:        { condition: service_healthy }
       kafka:          { condition: service_healthy }
       kibana:         { condition: service_healthy }
+      kubernetes:     { condition: service_healthy }
+      kube-state:     { condition: service_started }
       memcached:      { condition: service_healthy }
       mongodb:        { condition: service_healthy }
       mysql:          { condition: service_healthy }
@@ -96,6 +99,22 @@ services:
       file: ${ES_BEATS}/testing/environments/${TESTING_ENVIRONMENT}.yml
       service: kibana
 
+  kubernetes:
+    build: ${PWD}/module/kubernetes/_meta
+    network_mode: host
+    pid: host
+    privileged: true
+    volumes:
+      - /:/rootfs:ro
+      - /sys:/sys
+      - /var/lib/docker:/var/lib/docker
+      - /var/run:/var/run
+
+  kube-state:
+    build:
+      context: ${PWD}/module/kubernetes/_meta/
+      dockerfile: Dockerfile.kube-state
+
   memcached:
     build: ${PWD}/module/memcached/_meta