Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Watch only pods in state Running and Unknown #3744

Merged
merged 1 commit into from
Dec 7, 2020

Conversation

jbartosik
Copy link
Collaborator

Stop watching pods in Succeeded and Failed states. I ran full-vpa e2e tests on this and they passed (including the OOM test).

I grabbed VPA logs about OOMs and one was discarded. But it was because the OOM was too old, not because we removed the container (which was my worry about the change):

$ kubectl logs -n kube-system vpa-recommender-f7c794c46-swx8l|grep -i oom
I1204 14:57:45.110882       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 14:57:35 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-7787 PodName:hamster-69d67ccbd4-lghrd} ContainerName:hamster}}
I1204 14:57:45.110956       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 14:57:37 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-7787 PodName:hamster-69d67ccbd4-j6gsz} ContainerName:hamster}}
I1204 14:57:45.110981       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 14:57:37 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-7787 PodName:hamster-69d67ccbd4-967k8} ContainerName:hamster}}
I1204 14:59:45.111255       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 14:58:50 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-7787 PodName:hamster-69d67ccbd4-lghrd} ContainerName:hamster}}
I1204 14:59:45.111350       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 14:59:35 +0000 UTC Memory:1555165137 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-7787 PodName:hamster-69d67ccbd4-lm6km} ContainerName:hamster}}
I1204 15:01:45.113617       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:01:04 +0000 UTC Memory:262144000 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-h2z5s} ContainerName:hamster}}
W1204 15:01:45.113662       1 cluster_feeder.go:421] Failed to record OOM {Timestamp:2020-12-04 15:01:04 +0000 UTC Memory:262144000 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-h2z5s} ContainerName:hamster}}. Reason: error while recording OOM for {{vertical-pod-autoscaling-202 hamster-69d67ccbd4-h2z5s} hamster}, Reason: OOM event will be discarded - it is too old (2020-12-04 15:01:04 +0000 UTC)
I1204 15:01:45.113725       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:01:23 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-nn89m} ContainerName:hamster}}
I1204 15:01:45.113826       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:01:24 +0000 UTC Memory:1073741824 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-k2ggq} ContainerName:hamster}}
I1204 15:01:45.113936       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:01:23 +0000 UTC Memory:262144000 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-h2z5s} ContainerName:hamster}}
I1204 15:02:45.111330       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:01:53 +0000 UTC Memory:262144000 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-h2z5s} ContainerName:hamster}}
I1204 15:03:45.113214       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:02:35 +0000 UTC Memory:262144000 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-h2z5s} ContainerName:hamster}}
I1204 15:03:45.113262       1 cluster_feeder.go:419] OOM detected {Timestamp:2020-12-04 15:03:34 +0000 UTC Memory:1555165137 ContainerID:{PodID:{Namespace:vertical-pod-autoscaling-202 PodName:hamster-69d67ccbd4-jzc6d} ContainerName:hamster}}

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 4, 2020
@jbartosik
Copy link
Collaborator Author

/assign @bskiba

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Dec 4, 2020
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 4, 2020
Copy link
Contributor

@mwielgus mwielgus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 7, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jbartosik, mwielgus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 1b2473a into kubernetes:master Dec 7, 2020
@jbartosik jbartosik mentioned this pull request Jan 12, 2021
@jbartosik jbartosik deleted the get-only-running-pods branch January 15, 2021 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants