Skip to content

Latest commit

 

History

History
108 lines (86 loc) · 6.65 KB

metrics.md

File metadata and controls

108 lines (86 loc) · 6.65 KB
Raw

Driver Metrics

Prerequisites

  1. Install Prometheus Operator in your cluster:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack

  1. Enable metrics by configuring controller.enableMetrics and node.enableMetrics.

  2. Deploy EBS CSI Driver:

$ helm upgrade --install aws-ebs-csi-driver --namespace kube-system ./charts/aws-ebs-csi-driver --values ./charts/aws-ebs-csi-driver/values.yaml

Overview

Installing the Prometheus Operator and enabling metrics will deploy a Service object that exposes the EBS CSI Driver's controller metric port through a ClusterIP. Additionally, a ServiceMonitor object is deployed which updates the Prometheus scrape configuration and allows scraping metrics from the endpoint defined. For more information, see the manifest metrics.yaml

AWS API Metrics

The EBS CSI Driver will emit AWS API metrics to the following TCP endpoint: 0.0.0.0:3301/metrics if controller.enableMetrics: true has been configured in the Helm chart.

The metrics will appear in the following format:

aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.005"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.01"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.025"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.05"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.1"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.25"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="0.5"} 0
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="1"} 1
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="2.5"} 1
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="5"} 1
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="10"} 1
aws_ebs_csi_api_request_duration_seconds_bucket{request="AttachVolume",le="+Inf"} 1
aws_ebs_csi_api_request_duration_seconds_sum{request="AttachVolume"} 0.547694574
aws_ebs_csi_api_request_duration_seconds_count{request="AttachVolume"} 1
...

To manually scrape AWS metrics:

$ export ebs_csi_controller=$(kubectl get lease -n kube-system ebs-csi-aws-com -o=jsonpath="{.spec.holderIdentity}")
$ kubectl port-forward $ebs_csi_controller 3301:3301 -n kube-system
$ curl 127.0.0.1:3301/metrics

EBS Node Metrics

The EBS CSI Driver will emit container storage Interface managed devices metrics to the following TCP endpoint: 0.0.0.0:3302/metrics if node.enableMetrics: true has been configured in the Helm chart.

The metrics will appear in the following format:

nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.001"} 0
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.0025"} 0
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.005"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.01"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.025"} 1
nvme_collector_duration_seconds_bucket{instance_id="instance-id}",le="0.05"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.1"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.25"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="0.5"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="1"} 1
nvme_collector_duration_seconds_bucket{instance_id="instance-id}",le="2.5"} 1
nvme_collector_duration_seconds_bucket{instance_id="instance-id}",le="5"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="10"} 1
nvme_collector_duration_seconds_bucket{instance_id="{instance-id}",le="+Inf"} 1
...

To manually scrape AWS metrics:

$ kubectl port-forward $ebs_csi_node_pod_name 3302:3302 -n kube-system
$ curl 127.0.0.1:3302/metrics

Volume Stats Metrics

The EBS CSI Driver emits Kubelet mounted volume metrics for volumes created with the driver.

The following metrics are currently supported:

Metric name Metric type Description Labels
kubelet_volume_stats_capacity_bytes Gauge The capacity in bytes of the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>
kubelet_volume_stats_available_bytes Gauge The number of available bytes in the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>
kubelet_volume_stats_used_bytes Gauge The number of used bytes in the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>
kubelet_volume_stats_inodes Gauge The maximum number of inodes in the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>
kubelet_volume_stats_inodes_free Gauge The number of free inodes in the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>
kubelet_volume_stats_inodes_used Gauge The number of used inodes in the volume namespace=<persistentvolumeclaim-namespace>
persistentvolumeclaim=<persistentvolumeclaim-name>

For more information about the supported metrics, see VolumeUsage within the CSI spec documentation for the NodeGetVolumeStats RPC call.

For more information about metrics in Kubernetes, see the Metrics For Kubernetes System Components documentation.

CSI Operations Metrics

The csi_operations_seconds metrics reports a latency histogram of kubelet-initiated CSI gRPC calls by gRPC status code.

To manually scrape Kubelet metrics:

$ kubectl proxy
$ kubectl get --raw /api/v1/nodes/<insert_node_name>/proxy/metrics