Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-4.17] OCPBUGS-43820: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup #2126

Open
wants to merge 4 commits into
base: release-4.17
Choose a base branch
from
Open

[release-4.17] OCPBUGS-43820: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup #2126

wants to merge 4 commits into from
+111 −23

Conversation

haircommander
Copy link
Member

This is a manual cherry-pick of #2031

fixes verify test from #2123

none

@haircommander
UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from mi…
65330c1 
…ssing cpuset cgroup

on None cpumanager policy, cgroupv2, and systemd cgroup manager, kubelet
could get into a situation where it believes the cpuset cgroup was created
(by libcontainer in the cgroupfs) but systemd has deleted it, as it wasn't requested
to create it. This causes one unnecessary restart, as kubelet fails with

`failed to initialize top level QOS containers: root container [kubepods] doesn't exist.`

This only causes one restart because the kubelet skips recreating the cgroup
if it already exists, but it's still a bother and is fixed this way

adapted version of kubernetes@c51195d, avoid merge conflict
this can be dropped in the 4.19 rebase

Signed-off-by: Peter Hunt <pehunt@redhat.com>
@haircommander
UPSTREAM: <carry>: kubelet/cm: move CPU reading from cm to cm/cpumanager
40151eb 
Adapted from kubernetes@77d03e4 to avoid merge conflict
This can be dropped in the 4.19 rebase

Authored-by: Francesco Romani <fromani@redhat.com>
Signed-off-by: Peter Hunt <pehunt@redhat.com>
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Nov 1, 2024
@openshift-ci-robot
Copy link

@haircommander: This pull request references Jira Issue OCPBUGS-43820, which is invalid:

  • expected the bug to target either version "4.18." or "openshift-4.18.", but it targets "4.17.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #2031

fixes verify test from #2123

none

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Nov 1, 2024
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2024
@haircommander haircommander changed the base branch from master to release-4.17 November 1, 2024 20:31
@openshift-ci-robot
Copy link

@haircommander: the contents of this pull request could not be automatically validated.

The following commits are valid:

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@openshift-ci-robot
Copy link

@haircommander: This pull request references Jira Issue OCPBUGS-43820, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-28812 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

This is a manual cherry-pick of #2031

fixes verify test from #2123

none

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2024
ffromani and others added 2 commits November 1, 2024 16:35
@ffromani @haircommander
UPSTREAM: 125923: e2e_node: add a test to verify the kubelet starts
54e92e6 
with systemd cgroup driver and cpumanager none policy.

This was originally planned to be a correctness check for
https://issues.k8s.io/125923, but it was difficult to reproduce the bug,
so it's now a regression test against it.

Signed-off-by: Francesco Romani <fromani@redhat.com>
Signed-off-by: Peter Hunt <pehunt@redhat.com>
@haircommander
UPSTREAM: 125923: e2e_node: use restart instead of start stop
777f620 
Signed-off-by: Peter Hunt <pehunt@redhat.com>
@haircommander haircommander force-pushed the cherry-pick-2031-to-release-4.17 branch from 4cb7d5c to 777f620 Compare November 1, 2024 20:35
@openshift-ci-robot
Copy link

@haircommander: the contents of this pull request could not be automatically validated.

The following commits are valid:

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@haircommander haircommander mentioned this pull request Nov 1, 2024
Closed
@rphillips
Copy link

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 1, 2024
@mrunalp mrunalp added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. labels Nov 1, 2024
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 1, 2024

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: haircommander, mrunalp, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rphillips
Copy link

/label backport-risk-assessed

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 1, 2024
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 2, 2024 •
edited
Loading

@haircommander: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-downgrade 4cb7d5c link true /test e2e-aws-ovn-downgrade
ci/prow/k8s-e2e-gcp-serial 4cb7d5c link true /test k8s-e2e-gcp-serial
ci/prow/e2e-gcp 4cb7d5c link true /test e2e-gcp
ci/prow/e2e-aws-ovn-fips 4cb7d5c link true /test e2e-aws-ovn-fips
ci/prow/e2e-aws-crun-wasm 4cb7d5c link true /test e2e-aws-crun-wasm
ci/prow/e2e-gcp-ovn-upgrade 4cb7d5c link true /test e2e-gcp-ovn-upgrade
ci/prow/e2e-aws-ovn-crun 4cb7d5c link true /test e2e-aws-ovn-crun
ci/prow/k8s-e2e-gcp-ovn 4cb7d5c link true /test k8s-e2e-gcp-ovn
ci/prow/e2e-aws-ovn-cgroupsv2 4cb7d5c link true /test e2e-aws-ovn-cgroupsv2
ci/prow/artifacts 4cb7d5c link true /test artifacts

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@haircommander
Copy link
Member Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrunalp mrunalp mrunalp approved these changes

@deads2k deads2k Awaiting requested review from deads2k

Assignees

@rphillips rphillips

@kasturinarra kasturinarra

@zhouying7780 zhouying7780

@xingxingxia xingxingxia

@sunilcio sunilcio

@duanwei33 duanwei33

@wangke19 wangke19

@gangwgr gangwgr

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.