Ok so after a week of running this in production, I feel like either I missed some critical piece of information, or it works VERY differently from what it feels like it should.

An image is worth a thousand words, so:

Unless I'm mistaken, it doesn't just reserve memory, but also enforces the limits of the cgroups, including the system slice's.

So you can very well end up with OOMkills on your control plane nodes' etcd... despite plenty of memory being available (here in the section of time for the past week; plenty of free memory, at the cost of OOMkill'd etcds...):

While I appreciate this works in a similar fashion to k8s' memory limits the whole part of naming it system_reserved, system_master_memory_reserved, etc is... confusing to me.

My understanding (which was clearly wrong) was that those reservations only took away from the pods slice, rather than applying as hard limits...

Makes me a bit uneasy overall.

To note, my guess is that the "culprit" is actually kubelet_enforce_node_allocatable after reading https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#enforcing-node-allocatable closer

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

I agree, this seems to be a serious issue.

Before #9209 the default result in /etc/kubernetes/kubelet-config.yaml included

kubeReserved:
  cpu: 100m
  memory: 256Mi

Referring to official docs (capitalization mine for emphasis)

To OPTIONALLY enforce kubeReserved on kubernetes system daemons, specify the parent control group for kube
daemons as the value for kubeReservedCgroup setting, and add kube-reserved to enforceNodeAllocatable.

My interpretation is that defining resources for KubeReserved simply reduces the node allocatable resources by a small amount, to ensure that pods don't overwhelm a node. This is a simple scheduling matter: allocatable resources = (node resources) - (kube reserved) - (system reserved)
Preserving some node resources for critical daemons should be a default behaviour as it is very important for stability of production clusters.

As the doc notes, an optional additional step can be taken, to define a separate cgroup for k8s daemons to run in. But doing this, and actually creating that cgroup, aside from being optional and unnecessary, can be dangerous if it restricts the resource usage of k8s daemons.

In other words I think we need the following as default behaviour:

1. Some baseline kubeReserved resources should be specified by default (like a k8s resource request for important daemons).
2. Enforcement or limitation of the resource usage of important daemons can cause instability. So it seems to me a kubeReserved cGroup should NOT be specified by default (like a k8s resource limit that can cause important daemons to be killed).

As the doc notes, an optional additional step can be taken, to define a separate cgroup for k8s daemons to run in. But doing this, and actually creating that cgroup, aside from being optional and unnecessary, can be dangerous if it restricts the resource usage of k8s daemons.

That would explain the strange behaviors I ended up seeing indeed.

I sort of gave up and massively increased RAM capacity to work around it as I wasn’t able to come up with a sensible solution at the time…

But if having a kube system cgroup causes things like etcd to be OOMkilled from going over it, it is indeed more harmful than anything else.

@Tristan971 What value of kubelet_enforce_node_allocatable did you have? Based on the documentation it should only apply to pods by default, but it's a bit ambiguous because kubeReservedCgroup is described as "enforcing kubeReserved on kubernetes system daemons". So I'm not sure if kubeReservedCgroup is sufficient to explain the etcd OOM killing you noticed, or whether kubelet_enforce_node_allocatable would also have to include "kube-reserved".

What value of kubelet_enforce_node_allocatable did you have?

It appears that I also decided to not gamble on it, as I see the following diff in the commit where I changed it:

- kubelet_enforce_node_allocatable: "pods,kube-reserved,system-reserved"
+ kubelet_enforce_node_allocatable: "pods"

Additionally at the same time

- system_master_memory_reserved: 512Mi
+ system_master_memory_reserved: 4Gi

This was more than 1.5 years ago however, so I unfortunately cannot recall whether both were necessary or not or if I was simply trying to get the issue to go away.

kubelet_enforce_node_allocatable: "pods,kube-reserved,system-reserved" would explain the killing of the system daemons. The default is just pods.

#11367 will fix the issue of the disappearing reservation.

Thanks for the PR and pushing through with this! 👍