Enable collecting cri-o metrics

[ahrennsiva]

I can currently run:
minikube start --container-runtime=cri-o

This will start minikube with CRI-O as the container runtime. CRI-O exposes metrics with this flag enabled --enable-metrics: Enable metrics endpoint. Default is localhost:9090. I would like to access the metrics exposed here for minikube but this currently isn't possible without additional changes in the VM itself.

Would it be possible to allow passing in runtime flags as a feature, at least for all currently supported alternative container runtimes?

[afbjorklund]

Thanks for reporting! This seems reasonable, we might have to rename the docker-opt/docker-env

[afbjorklund]

Should probably enable the metrics by default, until we can improve upon the docker-centric situation

[afbjorklund]

Seems like we forgot all about this one... Should still modify the docker-centric variables.

But maybe we can just make --enable-metrics the default for crio, by hacking the service ?

i.e. install this, and uncomment the line:

# /etc/sysconfig/crio

# use "--enable-metrics" and "--metrics-port value"
#CRIO_METRICS_OPTIONS="--enable-metrics"

#CRIO_NETWORK_OPTIONS=
#CRIO_STORAGE_OPTIONS=

Currently we overwrite this with a custom version:

deploy/iso/minikube-iso/package/crio-bin/crio-bin.mk:   echo 'CRIO_OPTIONS="--log-level=debug"' > $(TARGET_DIR)/etc/sysconfig/crio

And the $CRIO_METRICS_OPTIONS are removed:

-ExecStart=/usr/local/bin/crio \
-          $CRIO_CONFIG_OPTIONS \
-          $CRIO_RUNTIME_OPTIONS \
-          $CRIO_STORAGE_OPTIONS \
-          $CRIO_NETWORK_OPTIONS \
-          $CRIO_METRICS_OPTIONS
+ExecStart=/usr/bin/crio \
+    $CRIO_OPTIONS \
+    $CRIO_MINIKUBE_OPTIONS

The custom generated minikube configuration:

       var (
                crioOptsTmpl = `
CRIO_MINIKUBE_OPTIONS='{{ range .EngineOptions.InsecureRegistry }}--insecure-registry {{.}} {{ end }}'
`
                crioOptsPath = "/etc/sysconfig/crio.minikube"
        )

[afbjorklund]

Tried to enable this in crio.conf, but it failed (to apply, since the config is being overwritten)

[afbjorklund]

Fixed now, actually changed in #5970 but stopped being overwritten with #6219

$ grep -C3 metrics /etc/crio/crio.conf
	"/opt/cni/bin/",
]

# A necessary configuration for Prometheus based metrics retrieval
[crio.metrics]

# Globally enable or disable metrics support.
enable_metrics = true

# The port on which the metrics server will listen.
metrics_port = 9090

$ curl localhost:9090/metrics   
# HELP container_runtime_crio_operations Cumulative number of CRI-O operations by operation type.
# TYPE container_runtime_crio_operations counter
container_runtime_crio_operations{operation_type="container_status"} 264
container_runtime_crio_operations{operation_type="create_container"} 7
container_runtime_crio_operations{operation_type="image_status"} 16
container_runtime_crio_operations{operation_type="list_containers"} 736
container_runtime_crio_operations{operation_type="list_images"} 53
container_runtime_crio_operations{operation_type="list_pod_sandbox"} 691
container_runtime_crio_operations{operation_type="pod_sandbox_status"} 18
container_runtime_crio_operations{operation_type="run_pod_sandbox"} 7
container_runtime_crio_operations{operation_type="start_container"} 7
container_runtime_crio_operations{operation_type="status"} 76
container_runtime_crio_operations{operation_type="version"} 59
...

Will be included in v1.7.0, sorry it took so long... (still have to fix docker-opt/docker-env)