Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Populate both CPU and Memory resource container metrics if one is specified #330

Merged
merged 1 commit into from
Oct 30, 2020

Conversation

joelsmith
Copy link
Contributor

We're experiencing an issue where HPA won't work with pods that have init containers. The root cause seems to be that the init containers have memory metrics but no CPU metrics, and HPA refuses to take action if any containers in the pod have container metrics, but are missing either CPU or memory.

This patch makes the adapter metrics look like the metrics server endpoint.

Before:

items:
- apiVersion: metrics.k8s.io/v1beta1
  containers:
  - name: gerbil
    usage:
      cpu: "0"
      memory: 1568Ki
  - name: gerbil-init
    usage:
      memory: "0"
  kind: PodMetrics

After:

items:
- apiVersion: metrics.k8s.io/v1beta1
  containers:
  - name: gerbil
    usage:
      cpu: "0"
      memory: 1568Ki
  - name: gerbil-init
    usage:
      cpu: "0"
      memory: "0"
  kind: PodMetrics

@s-urbaniak
Copy link
Contributor

lgtm, thank you for the contribution!

@s-urbaniak s-urbaniak merged commit 87c429b into kubernetes-sigs:master Oct 30, 2020
@s-urbaniak
Copy link
Contributor

@joelsmith general question though: we should recheck if metrics-server behaves the same and clarify if the behavior is unique across all resource metrics server implementations.

@joelsmith
Copy link
Contributor Author

@s-urbaniak while the implementation is different, metrics server does indeed provide both CPU and memory for the init container in this case. I verified on an OCP cluster by disabling the prometheus adapter and replacing it with metrics server. While metrics server does not have special code in it to populate missing CPU or memory values, the kubelet stats summary API does. Since the kubelet metrics add the missing metrics, then metrics server has them. Since the prometheus adapter doesn't use the stats summary API (instead getting metrics from prometheus with scrapes the values from the kubelet's cadvisor endpoint), it did not have the zero-valued CPU or memory metrics for init containers. Thanks for merging!

In case you are interested, here's the PR that we added some time ago to fix the issue in the metrics path that metrics server uses:
https://github.com/kubernetes/kubernetes/pull/88734/files#diff-89caf51d2d7b9fcbc06bf5c7c3aca05867afbbcfb8e1b4482d8237c7169335c5R45

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants