diff --git a/docs/walkthrough/scrape-promitor-promitor-with-prometheus-on-azure-kubernetes-service-using-managed-identity.md b/docs/walkthrough/scrape-promitor-promitor-with-prometheus-on-azure-kubernetes-service-using-managed-identity.md new file mode 100644 index 000000000..81dc0a16d --- /dev/null +++ b/docs/walkthrough/scrape-promitor-promitor-with-prometheus-on-azure-kubernetes-service-using-managed-identity.md @@ -0,0 +1,398 @@ +--- +layout: default +title: Deploying Promitor, Prometheus, and Grafana on an AKS Cluster +redirect_from: + - /walkthrough/ +--- + +## Introduction + +This walkthrough will allow you to deploy a full AKS cluster and connect your Prometheus instance to Azure Monitoring through Promitor, without having to manage and secure any sensitive password. + +This walkthrough is almost the same as [Deploying Promitor, Prometheus, and Grafana on an AKS Cluster](/scrape-promitor-with-prometheus-on-azure-kubernetes-service) but using a **managed identity** instead of a **service principal**. +AKS requires an identity to create additional resources (like load balancers, disks ...) +- Using a Service Principal requires you to provide an application id and an application key (password). +- Using a Managed Identity requires you to create the identity on Azure, but you don't need to provide any password. + +Check the official Microsoft documentation about [Managed Identity in Azure Kubernetes Service](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity). + +We will use the [aad-pod-identity](https://azure.github.io/aad-pod-identity/) project to configure identities for all your cluster pods, and rely on it for **Promitor**. + +## Table of Contents + +- **[Deploy Azure Infrastructure](#deploy-azure-infrastructure)** + - [Create a Resource Group](#create-a-resource-group) + - [Create a Service Principal](#create-a-service-principal) + - [Create a Service Bus Namespace and Queue](#create-a-service-bus-namespace-and-queue) + - [Create an AKS Cluster](#create-an-aks-cluster) +- **[TO COMPLETE]** + +## Prerequisites + +- [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest), to be able to deploy resources through the command line. +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/), the Kubernetes + command-line tool. It can also be installed via the Azure CLI with `az aks install-cli`. +- [Helm](https://helm.sh/docs/using_helm/#installing-the-helm-client), a Kubernetes + deployment manager. +- [WSL](https://docs.microsoft.com/en-us/windows/wsl/), if you are using a Windows machine to deploy your whole solution. + +## Deploy Azure Infrastructure + +### Preparing script + +Since we are going to use a lot of bash scripts with different variables values, it can be a good idea to parameterize everything. +So far, we can start by **exporting** all the values we need: + +```bash +# SUBSCRIPTION_ID represents the Azure subscription id you will use to access your Azure resources +export SUBSCRIPTION_ID= +# LOCATION represents the location where your cluster will be deployed +export LOCATION=northeurope +# CLUSTER_NAME represents the name of your AKS Cluster +export CLUSTER_NAME=PromitorCluster +# RG_NAME represents the resource groupe name where your cluster will be deployed +export RG_NAME=PromitorWithManagedIdentityRG +# AD_POD_IDENTITY_NAME represents the name of your future configured aad-pod-identity. Be careful, should be lower case alphanumeric characters, '-' or '.' +export AD_POD_IDENTITY_NAME=promitor-identity + + +# As an example, we are going to use a service bus from where we want to grab some metrics, through Promitor +# SERVICE_BUS_NAMESPACE represents the name of you service bus namespace. +# Be careful as Service Bus Namespaces need to be globally unique. +export SERVICE_BUS_NAMESPACE=PromitorUniqueNameServiceBus + +# SERVICE_BUS_QUEUE represents the name of you service bus queue. +export SERVICE_BUS_QUEUE=demo_queue +``` + + +### Create a Resource Group + +```bash +az group create --name $RG_NAME --location $LOCATION +``` + +Output: + +```json +{ + "id": "/subscriptions//resourceGroups/PromitorWithManagedIdentityRG", + "...": "..." +} +``` +### Create a Service Bus Namespace and Queue + +First we'll need to create a namespace. This **Service Bus Namespace** will output insights into Azure Monitoring. +These insights will be exposed to **Prometheus** through **Promitor**. + +_Consider this deployment as an example of an Azure resource we want to track from Prometheus._ + +```bash +az servicebus namespace create \ + --resource-group $RG_NAME \ + --name $SERVICE_BUS_NAMESPACE \ + --location $LOCATION +``` + +We'll then create a queue in that namespace: + +```bash +az servicebus queue create \ + --resource-group $RG_NAME \ + --namespace-name $SERVICE_BUS_NAMESPACE \ + --name $SERVICE_BUS_QUEUE +``` + +### Create an AKS Cluster + +Create an AKS cluster using the managed identity option: + +```bash +az aks create --resource-group $RG_NAME \ + --name $CLUSTER_NAME \ + --generate-ssh-keys \ + --node-count 1 \ + --enable-managed-identity +``` + +A successful cluster creation using managed identities contains this service principal profile information: + +```bash +"servicePrincipalProfile": { + "clientId": "msi" + } +``` + +## Cluster Setup + +### Get credentials + +Getting the credentials from your AKS cluster will allow you to use the `kubectl` CLI, locally. +You can get your cluster's credentials using this command: + +```bash +az aks get-credentials \ + --name $CLUSTER_NAME \ + --resource-group $RG_NAME +``` + +This will save these credentials to your kubeconfig file and set your new cluster +as your current context for all `kubectl` commands. + +Verify your credentials and check that your cluster is up and running with +`kubectl get nodes`. + +### Get AKS Managed Identity and MC resource group + +To be able to configure **aad-pod-identity**, we will need some information from your newly deployed **AKS** cluster: +- **MC Resource Group**: Resource group where the AKS internal resources have been deployed +- **AKS Managed Identity Id**: AKS Managed Identity created internally (since we used the `--enable-managed-identity` option) that is used by your AKS cluster to access Azure resources. + +```bash +echo "Retrieving MC resource group" +export mc_aks_group=$(az aks show -g $RG_NAME -n $CLUSTER_NAME --query nodeResourceGroup -otsv) + +echo "Retrieving cluster identity ID, which will be used for role assignment" +export mc_aks_mi_id="$(az aks show -g ${RG_NAME} -n ${CLUSTER_NAME} --query identityProfile.kubeletidentity.clientId -otsv)" +``` + +## AAD Pod Identity + +### Configure AKS Managed Identity for AAD Pod Identity + +If you want to know more about how to configure and manage your aad pod identities, check the official documentation: https://azure.github.io/aad-pod-identity/docs/ + +In this walkthrough we are going to configure the **AKS Managed Identity** to allow **aad-pod-identity** to access required resources. (More info: [AAD Pod Identity Role Assignements](https://azure.github.io/aad-pod-identity/docs/getting-started/role-assignment/#performing-role-assignments)) + +```bash +echo "Assigning role needed by Pod Identity:" +echo "Assigning 'Managed Identity Operator' role to ${mc_aks_mi_id} on resource group ${mc_aks_group}" +az role assignment create --role "Managed Identity Operator" --assignee "${mc_aks_mi_id}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/${mc_aks_group}" + +echo "Assigning 'Virtual Machine Contributor' role to ${mc_aks_mi_id} on resource group ${mc_aks_group}" +az role assignment create --role "Virtual Machine Contributor" --assignee "${mc_aks_mi_id}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/${mc_aks_group}" + +echo "Assigning 'Managed Identity Operator' role to ${mc_aks_mi_id} on resource group ${RG_NAME}" +az role assignment create --role "Managed Identity Operator" --assignee "${mc_aks_mi_id}" --scope "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/${RG_NAME}" +``` + +### Install AAD Pod Identity + +To deploy AAD Pod Identity, we need to add the chart: + +```bash +echo "Add helm repo for aad pod identity" +helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts +helm repo update +``` + +Then we are good to go to deploy the chart: + +```bash +echo "Install aad pod identity and allow use of kubenet" +helm install aad-pod-identity aad-pod-identity/aad-pod-identity --set nmi.allowNetworkPluginKubenet=true +``` + +### Create the Managed Identity for Promitor + +We can basically use the already deployed and created managed identity used by AKS, but since we want to [separate the concerns](https://en.wikipedia.org/wiki/Separation_of_concerns), +we are going to create a new managed identity that will be the identity used later by all your Pods. + +```bash +echo "Create identity $AD_POD_IDENTITY_NAME in resource group $RG_NAME" +az identity create -g ${RG_NAME} -n ${AD_POD_IDENTITY_NAME} +``` + +This identity will be used by **Promitor** to access **Azure Monitoring**. +So far, we need to allow this identity to read data from **Azure Monitoring** through **RBAC Roles Assignements**: + +> _**Warning:**_ It can take some times before the Identity is correctly propagated in Azure Active Directory. +> So far if you encountered an error where the identity is not found, please wait 60 sec and retry + +```bash +echo "Getting Azure Identity used by aad pod identity" +export AD_POD_IDENTITY_CLIENT_ID=$(az identity show -g ${RG_NAME} -n ${AD_POD_IDENTITY_NAME} --query "clientId" -o tsv) +export AD_POD_IDENTITY_RESOURCE_ID=$(az identity show -g ${RG_NAME} -n ${AD_POD_IDENTITY_NAME} --query "id" -o tsv) + +echo "Assign role 'Monitoring Reader' to identity $AD_POD_IDENTITY_NAME in resource group $RG_NAME" +az role assignment create --role "Monitoring Reader" --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG_NAME" --assignee "${AD_POD_IDENTITY_CLIENT_ID}" +``` + +_Note:_ You can check the role assignements using this command: +```bash +az role assignment list --assignee $AD_POD_IDENTITY_CLIENT_ID -g $RG_NAME | jq -r '.[].roleDefinitionName' +``` + +### Bind your Managed Identity to your Pods, through AAD Pod Identity + +The Managed Identity is created, AAD Pod Identity is deployed, it's time to bind your identity to your Pods: + +```bash +echo "Create an AzureIdentity in the AKS cluster that references the identity $AD_POD_IDENTITY_NAME" + +cat < _**Warning:**_ It can take some times to Aad Pod Identity to bind the identity to your deployed container. +> If you encountered an error, relaunch the `az login -i --debug` command after 60 sec. + +```bash +# Once you are log in the container, and have the bash command line available, try to login using the Managed Identity: +If you don t see a command prompt, try pressing enter. +bash-5.0# az login -i --debug +# You should have a line with 'MSI: token was retrieved. Now trying to initialize local accounts'" +``` + +If your Azure CLI container is able to retrieve some information from Azure +without having to log in with your credentials, +it means the CLI is using your Managed Identity, through the Pod Identity Binding. + +You may have a result indicating you are log in, using a **System Assigned Identity** + +```json +"user": { + "assignedIdentityInfo": "MSI", + "name": "systemAssignedIdentity", +} +``` + +## Deploy Promitor and Prometheus + +### Create a metrics declaration for Promitor + +Before deploying Promitor, you'll need a values file with the managed identity option & a metric declaration +file (these can also be the same file for ease of use). +The yaml below will scrape +one metric, queue length, from the queue created above. + +```yaml +azureMetadata: + tenantId: + subscriptionId: + resourceGroupName: + mode: "systemAssignedManagedIdentity" + aadpodidbinding: +metricDefaults: + aggregation: + interval: 00:05:00 + scraping: + schedule: "* * * * *" +metrics: + - name: demo_queue_size + description: "Amount of active messages of the 'demo_queue' queue" + resourceType: ServiceBusNamespace + azureMetricConfiguration: + metricName: ActiveMessages + aggregation: + type: Total + resources: + - namespace: + queueName: +``` + +### Deploy Promitor to your cluster using Helm + +To deploy, we'll first add the Promitor chart repository to helm: + +```bash +helm repo add promitor https://charts.promitor.io/ +helm repo update +``` + +With this repository added, we can deploy Promitor: + +```bash +helm install promitor-agent-scraper promitor/promitor-agent-scraper \ + --values your/path/to/metric-declaration.yaml +``` + +### Optional: Check your Promitor deployment + +You can check that Promitor is getting insights from you service bus, using the managed identity, with this commands: + +```bash +# Get promitor pod +export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/instance=promitor-agent-scraper" -o jsonpath="{.items[0].metadata.name}") + +kubectl port-forward --namespace default $POD_NAME 8080:88 +``` + +Now browse to the address http://127.0.0.1:8080/metrics and check your metrics are scrapped: + +``` html +# HELP demo_queue_size Amount of active messages of the 'demo_queue' queue +# TYPE demo_queue_size gauge +demo_queue_size{resource_group="ammdocs",subscription_id="xxxxx-xxxxx-xxxxx-xxxxxx-xxxxx",resource_uri="subscriptions/xxxxx-xxxxx-xxxxx-xxxxxx-xxxxx/resourceGroups/YOUR_RESOURCE_GROUP_NAME/providers/Microsoft.ServiceBus/namespaces/YOUR_SERVICE_BUS_NAMESPACE",instance_name="INSTANCE_NAME",entity_name="YOUR_SERVICE_BUS_QUEUE"} 0 1612952581417 +``` + +### Install Prometheus, Grafana and Prometheus Operator + +We are going to use the [prometheus-community stack chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) from the [Prometheus Monitoring Community](https://github.com/prometheus-community) to deploy a full Prometheus environment, already configured for Kubernetes. + +``` bash +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +``` + +```bash +cat > promitor-scrape-config.yaml <