Pause containers do not have "io.kubernetes.container.name" set to "POD"

[lbernail]

Hello

I'm not sure if this is really an issue but pause containers do not have an "io.kubernetes.container.name" label

For other runtimes (I checked for docker and crio) this label is set to "POD" and this appears in the kubelet cadvisor metrics for pod level metrics (network in particular).

Looks like we could add this label to sandboxLabels in RunPodSandbox

What do you think?

[Random-Liu]

We use something else for containerd pod network stats in cadvisor: https://github.com/google/cadvisor/blob/master/container/containerd/handler.go#L166.

The io.kubernetes.container.name label on pause container is a runtime implementation detail. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go#L585 The label is not applied by kubelet. Actually even if the label is applied by kubelet, we don't recommend people to rely on that, because that is not a guaranteed api, and can change in the future.

As for in containerd whether we should have it, I'm leaning to not adding unnecessary labels. But if we really need it, we can add it.

Is it easy to support containerd without the container name label on pause container in datadog? If yes, can we keep this issue open to collect more feedback? If there are many people already rely on this, we'll have to support it anyway.

@yujuhong Any opinion about this?

[yujuhong]

Yes, as @Random-Liu said, kubelet does not set this label. We set it in dockershim for backward-compatibility reason during transition. In principle, the namespace of the label key (`io.kubernetes.container.name) is owned by kubelet, and runtime should not use it at all.

If the monitoring agents rely on these labels heavily (regardless whether that's the best practice or not), we can consider letting kubelet apply a label to PodSandboxes so that it will be consistent across CRI implementations. The label key will likely not to be io.kubernetes.container.name, but something more pod/sandbox related.

[lbernail]

Thank you for your quick answers

In our case, we noticed it because pod level metrics on the kubelet /metrics/cadvisor looked like:

container_network_transmit_bytes_total{container_name="",id="/kubepods/burstable/xxx",image="k8s.gcr.io/pause:3.1",interface="eth0",name="xxx",namespace="xxx",pod_name="xxx"} 2.20063896e+08

When with other runtimes we had:

container_network_transmit_bytes_total{container_name="POD",id="/kubepods/burstable/xxx",image="k8s.gcr.io/pause:3.1",interface="eth0",name="xxx",namespace="xxx",pod_name="xxx"} 2.20063896e+08

It definitely makes sense for the container_name label to be empty in that case but we were surprised by the different behavior.

We already have a fix for the Datadog agent: instead of only identifying pod-level metrics by testing for container_name==POD we now also test for container_name=="" and pod_name!="" to support all runtimes.

[Random-Liu]

@lbernail Thanks a lot for reporting this!

If you already have a fix, let's keep the issue open and see whether there are other people need this behavior. If there are many, let's add it. If not, I prefer not adding it for now, because it is a legacy thing that we want to get rid of gradually.