Revise the page to accurately document the well-known labels, annotations and taints #25998
Conversation
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
sig/docs
|
Deploy preview for kubernetes-io-master-staging ready! Built with commit c6b704e https://deploy-preview-25998--kubernetes-io-master-staging.netlify.app |
| @@ -14,6 +14,8 @@ This document serves both as a reference to the values and as a coordination poi | |||
|
|
|||
| <!-- body --> | |||
|
|
|||
| # Labels | |||
There was a problem hiding this comment.
We do not use H1 in any pages. The "top-level" headers are always H2, i.e. ##.
|
|
||
| Used on: Service | ||
|
|
||
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. |
There was a problem hiding this comment.
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. | |
| The label is the tag name we use to differentiate multiple Services. Used currently for ELBs only. |
There was a problem hiding this comment.
I'd write:
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. | |
| Kubernetes uses this label to differentiate multiple Services. Used currently for ELBs only. |
however:
I also recommend explaining ELB, or rephrasing so that this text is not needed.
|
|
||
| Example: `experimental.windows.kubernetes.io/isolation-type: "hyperv"` | ||
|
|
||
| The annotation is used to run windows containers with hyperv isolation. |
There was a problem hiding this comment.
| The annotation is used to run windows containers with hyperv isolation. | |
| The annotation is used to run Windows containers with Hyper-V isolation. |
There was a problem hiding this comment.
I presume this is an annotation for a Pod. Is it a hint or something stronger? What acts on it (eg the kubelet)?
|
|
||
| ## experimental.windows.kubernetes.io/isolation-type | ||
|
|
||
| Example: `experimental.windows.kubernetes.io/isolation-type: "hyperv"` |
There was a problem hiding this comment.
Can we also add "used on:" for annotations, if possible?
There was a problem hiding this comment.
It also applies to the annotations, I will update it.
|
|
||
| Example: `ingressclass.kubernetes.io/is-default-class: "true"` | ||
|
|
||
| When a single IngressClass resource has this annotation set to true, new Ingress resource without a class specified will be assigned this default class. |
There was a problem hiding this comment.
| When a single IngressClass resource has this annotation set to true, new Ingress resource without a class specified will be assigned this default class. | |
| When a single IngressClass resource has this annotation set to `"true"`, new Ingress resource without a class specified will be assigned this default class. |
Boolean values are always a headache. So please accurately specify that the value is actually a string.
|
|
||
| Example: `node.kubernetes.io/unschedulable:NoSchedule` | ||
|
|
||
| The taint will be added when node becomes unschedulable and removed when node becomes scheduable. |
There was a problem hiding this comment.
Maybe add "This taint is added to a Node during node initialization to avoid race condition."
|
|
||
| Example: `node.kubernetes.io/memory-pressure:NoSchedule` | ||
|
|
||
| The taint will be added when node has memory pressure and removed when node has enough memory. |
There was a problem hiding this comment.
Maybe add "The kubelet detects memory pressure based on the memory.available and allocatableMemory.available observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed."
|
|
||
| Example: `node.kubernetes.io/disk-pressure:NoSchedule` | ||
|
|
||
| The taint will be added when node has disk pressure and removed when node has enough disk. |
There was a problem hiding this comment.
Maybe add "The kubelet detects disk pressure by observing the imagefs.available,
imagefs.inodesFree, nodefs.available, and nodefs.inodesFree(Linux only) metrics for a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the node condition and taint should be added/removed."
|
|
||
| Example: `node.kubernetes.io/network-unavailable:NoSchedule` | ||
|
|
||
| The taint will be added when node's network is unavailable and removed when network becomes ready. |
There was a problem hiding this comment.
Maybe add "This is initially set by the kubelet when the cloud provider used indicates a requirement for additional network configuration. Only when the route on the cloud is configured properly will the taint be removed by the cloud provider."
|
|
||
| Example: `node.kubernetes.io/pid-pressure:NoSchedule` | ||
|
|
||
| The taint will be added when there are not enough process numbers to assign on the node and removed when there are enough process numbers to assign on the node. |
There was a problem hiding this comment.
Maybe add "The kubelet checks the PIDs consumed by Kubernetes on a node and the rlimit setting on the node. The number of available PIDs are referred to as the pid.available metric. The metric is then compared to the corresponding threshold that can be set on the kubelet to determine if the node condition and taint should be added/removed."
There was a problem hiding this comment.
Does the kubelet also check the size of /proc/sys/kernel/pid_max when measuring for this taint? If so, we might want document that either here or on a page that we link to.
(if it's looking complex, that's a strong hint to note it and move on - we can sort out the details in a separate issue).
There was a problem hiding this comment.
Hi @CKchen0726
I really like the intent of this PR. It seems to me that more changes are needed before this could be ready to merge.
I'm very much OK with the idea of merging this even with a few snags left in; as I see it right now though, there are several details worth cleaning up first.
If you'd like advice on next steps, please feel free to reply.
|
|
||
| Used on: Service | ||
|
|
||
| The label is added by Controller to an Endpoint denoting if its parent Service is Headless. |
There was a problem hiding this comment.
| The label is added by Controller to an Endpoint denoting if its parent Service is Headless. | |
| The control plane adds this label to an Endpoints object when the owning Service is headless. |
BTW what about an EndpointSlice? Does the control plane also label EndpointSlices for headless services?
|
|
||
| Used on: Node | ||
|
|
||
| Starting in v1.17, this label is automatically added to adapt to the situation that the same cluster uses multiple versions of windows server. |
There was a problem hiding this comment.
| Starting in v1.17, this label is automatically added to adapt to the situation that the same cluster uses multiple versions of windows server. | |
| When the kubelet is running on Microsoft Windows, the kubelet automatically labels its node | |
| to record the version of Windows Server in use. |
Mentioning v1.17 isn't needed; this PR targets documentation for the current (v1.20) release.
|
|
||
| The label is added by Controller to an Endpoint denoting if its parent Service is Headless. | ||
|
|
||
| See [service.kubernetes.io/headless](#servicekubernetesioheadless). |
There was a problem hiding this comment.
| See [service.kubernetes.io/headless](#servicekubernetesioheadless). | |
| See [`service.kubernetes.io/headless`](#servicekubernetesioheadless). |
|
|
||
| Used on: Service | ||
|
|
||
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. |
There was a problem hiding this comment.
I'd write:
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. | |
| Kubernetes uses this label to differentiate multiple Services. Used currently for ELBs only. |
however:
I also recommend explaining ELB, or rephrasing so that this text is not needed.
|
|
||
| The label is the tag name we use to differentiate multiple services. Used currently for ELBs only. | ||
|
|
||
| See [kubernetes.io/service-name](#kubernetesioservice-name). |
There was a problem hiding this comment.
I don't understand this link.
|
|
||
| When upgrading to v1.17 with a cluster with EndpointSlice enabled, the `endpointslice.kubernetes.io/managed-by` label needs to be set on each EndpointSlice. | ||
|
|
||
| See [endpointslice.kubernetes.io/managed-by](#endpointslicekubernetesiomanaged-by). |
There was a problem hiding this comment.
I don't understand this link.
|
|
||
| The label can be set to true on an Endpoint resource to indicate that the EndpointSliceMirroring controller should not mirror this resource with EndpointSlices. | ||
|
|
||
| See [endpointslice.kubernetes.io/skip-mirror](#endpointslicekubernetesioskip-mirror). |
There was a problem hiding this comment.
I don't understand this link.
|
|
||
| Used on: Service | ||
|
|
||
| The label indicates that a custom service proxy will implement this service, kube-proxy does not generate iptables for the service. |
There was a problem hiding this comment.
Bear in mind that kube-proxy might not be running in iptables mode. Can we reword to be more technology-neutral?
|
|
||
| The label indicates that a custom service proxy will implement this service, kube-proxy does not generate iptables for the service. | ||
|
|
||
| See [service.kubernetes.io/service-proxy-name](#servicekubernetesioservice-proxy-name). |
There was a problem hiding this comment.
I don't understand this link.
|
|
||
| Example: `alpha.kubernetes.io/provided-node-ip: "10.0.0.1"` | ||
|
|
||
| The annotation is a node IP annotation set by the "external" cloud provider. |
There was a problem hiding this comment.
Is this right?
| The annotation is a node IP annotation set by the "external" cloud provider. | |
| The kubelet can set this annotation on a Node to denote its configured IPv4 address. |
(does it also work with IPv6? What about a dual-stack node?)
There was a problem hiding this comment.
this taint is only used for IPv4 address.
|
|
||
| Example: `node.kubernetes.io/unreachable:NoExecute` | ||
|
|
||
| The control plane monitors the health of all Nodes and decides whether to add or remove the taint. |
There was a problem hiding this comment.
pls consider my previous suggestions on this and the 'not-ready' labels. It is not clear what 'unreachable' means and what 'not-ready' means.
Your original text should be retained:
The taint will be added when node becomes unreachable (corresponding to Node's
Readycondition statusUnknown) and removed when node becomes reachable (Node'sReadycondition statusTrue).
|
|
||
| Example: `node.kubernetes.io/unreachable:NoExecute` | ||
|
|
||
| The node controller adds the taint to a node corresponding to the NodeCondition `Ready` being `Unknown`. |
There was a problem hiding this comment.
pls be careful when putting new terms. E.g. NodeCondition makes sense to developers, but not to average users.
|
|
||
| When kubelet is started with the "external" cloud provider, it sets this annotation on the Node to denote an IP address set from the command line flag (`--node-ip`). This IP is verified with the cloud provider as valid by the cloud-controller-manager. | ||
|
|
||
| ## The taints listed below are always used on Nodes. |
There was a problem hiding this comment.
| ## The taints listed below are always used on Nodes. | |
| **The taints listed below are always used on Nodes** |
|
|
||
| Example: `node.kubernetes.io/not-ready:NoExecute` | ||
|
|
||
| Sets the taint with `NotReady` on a node at creation. Otherwise, a new node may receive the taint with some delay causing Pods to be scheduled on a not-ready node. node controller will remove the taint when the node becomes ready. |
There was a problem hiding this comment.
| Sets the taint with `NotReady` on a node at creation. Otherwise, a new node may receive the taint with some delay causing Pods to be scheduled on a not-ready node. node controller will remove the taint when the node becomes ready. | |
| The node controller detects whether a node is ready by monitoring its health | |
| and adds or removes this taint accordingly. |
|
|
||
| Example: `node.kubernetes.io/not-ready:NoExecute` | ||
|
|
||
| The node controller detects whether a node is ready by monitoriing its health and adds or removes this taint accordingly. |
There was a problem hiding this comment.
| The node controller detects whether a node is ready by monitoriing its health and adds or removes this taint accordingly. | |
| The node controller detects whether a node is ready by monitoring its health and adds or removes this taint accordingly. |
|
|
||
| Example: `node.kubernetes.io/unreachable:NoExecute` | ||
|
|
||
| The node controller adds the taint to a node corresponding to the [NodeCondition](/docs/concepts/architecture/nodes/#condition) `Ready` being `Unknown`. |
There was a problem hiding this comment.
| The node controller adds the taint to a node corresponding to the [NodeCondition](/docs/concepts/architecture/nodes/#condition) `Ready` being `Unknown`. | |
| The node controller adds the taint to a node corresponding to the [Node condition](/docs/concepts/architecture/nodes/#condition) `Ready` being `Unknown`. |
|
|
||
| Example: `node.cloudprovider.kubernetes.io/uninitialized:NoSchedule` | ||
|
|
||
| Sets this taint on a node to mark it as unusable, when kubelet is started with the "external" cloud provider, until a controller from the cloud-controller-manager initializes this node, and then removes the taint. |
There was a problem hiding this comment.
The Kubelet's --cloud-provider options is now marked as deprecated, I have no idea if this parameter is appropriate.
|
|
||
| {{< note >}} Starting in v1.18, this annotation is deprecated in favor of [spec.ingressClassName]. {{< /note >}} | ||
|
|
||
| ## alpha.kubernetes.io/provided-node-ip |
There was a problem hiding this comment.
I think it do not require enabling alpha feature.
|
|
||
| ## service.kubernetes.io/headless {#servicekubernetesioheadless} | ||
|
|
||
| Example: `service.kubernetes.io/headless=""` |
There was a problem hiding this comment.
I found that the others have double quotes, keep the style consistent.
k8s-ci-robot
added
the
needs-rebase
k8s-ci-robot
removed
the
needs-rebase
|
|
||
| Used on: Service | ||
|
|
||
| Kubernetes uses this label to differentiate multiple Services. Used currently for `ELB`(Elastic Load Balancer) only. |
There was a problem hiding this comment.
Is this correct:
| Kubernetes uses this label to differentiate multiple Services. Used currently for `ELB`(Elastic Load Balancer) only. | |
| Kubernetes uses this label to differentiate multiple Services. Only used for Services that integrate with AWS Elastic Load Balancing using the legacy AWS cloud-controller-manager. |
?
|
|
||
| ## kubernetes.io/service-name {#kubernetesioservice-name} | ||
|
|
||
| Example: `kubernetes.io/service-name="nginx"` |
There was a problem hiding this comment.
Because nginx is (can be) a load-balancing solution, I suggest:
| Example: `kubernetes.io/service-name="nginx"` | |
| Example: `kubernetes.io/service-name="mywebapp"` |
Is that better?
|
|
||
| Used on: Pod | ||
|
|
||
| The annotation is used to run Windows containers with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated container, the kubelet should be started with feature gates HyperVContainer=true and the Pod should include the annotation experimental.windows.kubernetes.io/isolation-type=hyperv. |
There was a problem hiding this comment.
I'd write:
| The annotation is used to run Windows containers with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated container, the kubelet should be started with feature gates HyperVContainer=true and the Pod should include the annotation experimental.windows.kubernetes.io/isolation-type=hyperv. | |
| The annotation is used to run Windows pods with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated pod, the kubelet should be started with the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `HyperVContainer` enabled, and the Pod should include the annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`. |
|
|
||
| Used on: IngressClass | ||
|
|
||
| When a single IngressClass resource has this annotation set to `"true"`, new Ingress resource without a class specified will be assigned this default class. |
There was a problem hiding this comment.
| When a single IngressClass resource has this annotation set to `"true"`, new Ingress resource without a class specified will be assigned this default class. | |
| When exactly one IngressClass resource has this annotation set to `"true"`, and you then create a new Ingress resource without a class specified, the control plane assigns this default class to the new Ingress. |
?
|
|
||
| Example: `node.kubernetes.io/disk-pressure:NoSchedule` | ||
|
|
||
| The kubelet detects disk pressure based on `imagefs.available`, `imagefs.inodesFree`, `nodefs.available` and `nodefs.inodesFree`(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. |
There was a problem hiding this comment.
nit:
| The kubelet detects disk pressure based on `imagefs.available`, `imagefs.inodesFree`, `nodefs.available` and `nodefs.inodesFree`(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. | |
| The kubelet detects disk pressure based on `imagefs.available`, `imagefs.inodesFree`, `nodefs.available` and `nodefs.inodesFree` (Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. |
|
|
||
| Example: `node.kubernetes.io/pid-pressure:NoSchedule` | ||
|
|
||
| The kubelet checks D-value of the size of `/proc/sys/kernel/pid_max` and the PIDs consumed by Kubernetes on a node to get the number of available PIDs that referred to as the `pid.available` metric. The metric is then compared to the corresponding threshold that can be set on the kubelet to determine if the node condition and taint should be added/removed. |
There was a problem hiding this comment.
“D-value”? This might well confuse readers - it confuses me.
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 5ea4a1c81f9e569fd984421b844764a0a1ff95a7
|
|
/approve This overall looks good to me. Language tweakings can be left to future PRs. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tengqm The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
@CKchen0726 thanks - this really improves the usefulness of the page. |
Ref #25860