From 9beb796027f961dd30f4250220183490af2575ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Constan=C3=A7a=20Manteigas?= <113898685+constanca-m@users.noreply.github.com> Date: Tue, 18 Oct 2022 14:03:02 +0200 Subject: [PATCH] Update README.md for Metricbeat on K8s (#33308) * Update README.md for Metricbeat on K8s * Update README.md for Metricbeat on K8s * Clarified README.md for Metricbeat on K8s * Deleted HTML in README.md for Metricbeat on K8s * Apply suggestions from code review * Align table * Align table Co-authored-by: Giuseppe Santoro --- CHANGELOG.next.asciidoc | 2 +- deploy/kubernetes/metricbeat/README.md | 194 ++++++++++++++++++++++--- 2 files changed, 175 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index fde2c4273948..c75a5c8a27d0 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -149,7 +149,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff] *Metricbeat* - +- Update README file on how to run Metricbeat on Kubernetes. {pull}33308[33308] *Packetbeat* diff --git a/deploy/kubernetes/metricbeat/README.md b/deploy/kubernetes/metricbeat/README.md index 05ff876575f7..b8ef7aec71c0 100644 --- a/deploy/kubernetes/metricbeat/README.md +++ b/deploy/kubernetes/metricbeat/README.md @@ -2,30 +2,184 @@ ## Ship metrics from Kubernetes to Elasticsearch -### Kubernetes DaemonSet +### Settings -By deploying metricbeat as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) -we ensure we get a running metricbeat daemon on each node of the cluster. -Everything is deployed under `kube-system` namespace, you can change that by -updating YAML manifests under this folder. +We use official [Metricbeat Docker images](https://www.docker.elastic.co/r/beats/metricbeat), as they allow external files' configuration. Our YAML manifests are the following: -### Settings +* [metricbeat-daemonset-configmap.yaml](metricbeat-daemonset-configmap.yaml) to create the [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) necessary to [metricbeat-daemonset.yaml](metricbeat-daemonset.yaml). + +* [metricbeat-daemonset.yaml](metricbeat-daemonset.yaml) to create a [DeamonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). This way we ensure we get a running metricbeat daemon on each node of the cluster. +This file uses a set of environment variables to configure Elasticsearch output. We have two different sets: + * [Self-managed](https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html) Elasticsearch service: + + | Variable | Default | Description | + |------------------------|---------------|--------------------------------------| + | ELASTICSEARCH_HOST | elasticsearch | Elasticsearch host | + | ELASTICSEARCH_PORT | 9200 | Elasticsearch port | + | ELASTICSEARCH_USERNAME | elastic | Elasticsearch username for HTTP auth | + | ELASTICSEARCH_PASSWORD | changeme | Elasticsearch password | + + * Elasticsearch service on [Elastic Cloud](https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html): + + | Variable | Default | Description | + |--------------------|---------|--------------------| + | ELASTIC_CLOUD_ID | | Elastic cloud ID | + | ELASTIC_CLOUD_AUTH | | Elastic cloud auth | + +* [metricbeat-role-binding.yaml](metricbeat-role-binding.yaml),[metricbeat-role.yaml](metricbeat-role.yaml) and [metricbeat-service-account.yaml](metricbeat-service-account.yaml) to define a user's permissions in our namespace (more information on this can be found [here](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)). Notice that the namespace we are using is `kube-system`, but this can be changed by updating the YAML manifests. + + + +### Example + +In this example, we will explore two different options to run our Kubernetes cluster: + +- [For option 1](#Option-1.-Run-locally-on-Kind): Run locally using [Kind](https://kind.sigs.k8s.io/). +- [For option 2](#Option-2.-Run-on-GKE): Run on cloud using [GKE](https://cloud.google.com/kubernetes-engine). + +In both options, our cluster will have more than one node. +This way we can see the usage of our Metricbeat DaemonSet. + + +#### Prerequisites + +- [Kubectl](https://kubernetes.io/docs/tasks/tools/) to run commands against Kubernetes clusters. +- [For option 1](#Option-1.-Run-locally-on-Kind): [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) for running local Kubernetes clusters. +- [For option 2](#Option-2.-Run-on-GKE): [gcloud Cli](https://cloud.google.com/sdk/docs/install). + +#### Option 1. Run locally on Kind + +The first thing we need to do is create a Kubernetes cluster. On kind, the default [configuration](https://kind.sigs.k8s.io/docs/user/configuration/) of a cluster generates only one node. As we are using a DaemonSet to run our Metricbeat, we will make a cluster run 4 nodes, so we can check our pods are running on all (or some) of them. To do this, we simply create the file `cluster-config.yaml`: +```YAML +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: +- role: control-plane +- role: worker +- role: worker +- role: worker +``` + +Note that you can also specify your cluster's [nodes images](https://hub.docker.com/r/kindest/node/tags). + +We now can create a cluster using the command below. + +``` +$ kind create cluster --config=cluster-config.yaml +``` + + + +Notice that our cluster was created using the configuration shown above. The name of our cluster is the default one, `kind`. To interact with the cluster, you only need to specify the cluster name as a context in kubectl: + +``` +$ kubectl cluster-info --context kind-kind +``` + + +#### Option 2. Run on GKE + +1. Create a GKE cluster. You can find information on how to do that [here](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster). + +2. Connect to the newly created cluster. You can do that by running the following: +``` +$ gcloud container clusters get-credentials --zone --project +``` + + +#### Run Metricbeat on Kubernetes + +You can check the nodes of the newly created cluster by running the following: + +``` +$ kubectl get nodes +``` + +Metricbeat gets some metrics from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). +Clone the repository locally: + +``` +$ git clone git@github.com:kubernetes/kube-state-metrics.git +``` + +After that, you can deploy Kube State Metrics by running: + +``` +$ kubectl apply -f kube-state-metrics/examples/standard +``` + +To deploy Metricbeat to Kubernetes, clone this repository: + +``` +$ git clone git@github.com:elastic/beats.git +``` + +Don't forget to set the docker image version inside the [metricbeat-daemonset.yaml](metricbeat-daemonset.yaml). +Remember that you can find [Metricbeat Docker images here](https://www.docker.elastic.co/r/beats/metricbeat). +After that, run: + +``` +$ kubectl apply -f beats/deploy/kubernetes/metricbeat +``` + +You can check the pods of your cluster in the namespace `kube-system`. For that, you just run: + +``` +$ kubectl get pods -o wide --namespace=kube-system +``` + +You should be able to see each pod's node. +If you are using Linux/Unix/macOS you can request all pods whose names start with metricbeat: + +``` +$ kubectl get pods -o wide --namespace=kube-system | grep ^metricbeat +``` + + +#### Visualizing data in Kibana + +To visualize your data in Kibana, we will use Elastic Cloud. For that do the following: + + + +1. [Log in](https://cloud.elastic.co/home) to your Elastic Cloud account. + +2. Create a [deployment](https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html). + While waiting, you are prompted to save the admin credentials for your deployment which provides you with superuser access to Elasticsearch. + +3. On the deployment overview page, copy down the Cloud ID. + +4. Set the environment variables on the [metricbeat-daemonset.yaml](metricbeat-daemonset.yaml) file: + + ```YAML + - name: ELASTIC_CLOUD_ID + value: + - name: ELASTIC_CLOUD_AUTH + value: : + ``` + + If you already applied all the YAML manifest files you can just update this one: + + ``` + $ kubectl apply -f beats/deploy/kubernetes/metricbeat/metricbeat-daemonset.yaml + ``` + + Otherwise, make sure to apply all: + + ``` + $ kubectl apply -f beats/deploy/kubernetes/metricbeat + ``` + +5. Navigate to the **Analytics** endpoint and select **Discover**. To see Metricbeat data, make sure the predefined `metricbeat-*` index pattern is selected. + +6. To load a Kibana dashboard, you have to select **Dashboard** on the side menu. +For example, you can see data about your Metricbeat DaemonSet on the dashboard *[Metricbeat Kubernetes] DaemonSets*. +If by any chance, you don't have dashboards downloaded in Kibana, you can go [here](https://www.elastic.co/guide/en/beats/metricbeat/current/load-kibana-dashboards.html) to find a solution for that. -We use official [Beats Docker images](https://github.com/elastic/beats-docker), -as they allow external files configuration, a [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) -is used for kubernetes specific settings. Check [metricbeat-daemonset-configmap.yaml](metricbeat-daemonset-configmap.yaml) -for details. -Also, [metricbeat-daemonset.yaml](metricbeat-daemonset.yaml) uses a set of environment -variables to configure Elasticsearch output: +You should now be able to visualize your data on Kibana. -Variable | Default | Description --------- | ------- | ----------- -ELASTICSEARCH_HOST | elasticsearch | Elasticsearch host -ELASTICSEARCH_PORT | 9200 | Elasticsearch port -ELASTICSEARCH_USERNAME | elastic | Elasticsearch username for HTTP auth -ELASTICSEARCH_PASSWORD | changeme | Elasticsearch password +#### Visualizing your Kubernetes cluster -If there is an existing `elasticsearch` service in the kubernetes cluster these -defaults will use it. +If you are curious about your cluster, you can use [this](https://k8slens.dev/) to visualize resources.