kubeadm: not use CRI socket annotation on the Node object

[pacoxu]

Enhancement Description

• One-line enhancement description (can be used as a release note): in theory we don't need the CRI socket annotation on the Node object that kubeadm does any more.
• Kubernetes Enhancement Proposal:
• Discussion Link: do not annotate the Node CRI socket in the "init upload-config" phase kubeadm#1924
• Primary contact (assignee): pacoxu
• Responsible SIGs: cluster-lifecycle
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.29
  • Stable release target (x.y): 1.31
• Alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Key concerns that why this needs a KEP in kubernetes/kubeadm#1924 (comment)

• Users can use any CRI on different hosts, and the endpoint would be written on the Node object. If the CRI field is global in KubeletConfiguration managed by kubeadm, user settings during kubeadm join must override downloaded global configuration before being written to disk for kubelet consumption with --config. Kubeadm upgrade re-downloads KubeletConfiguration from cluster on each upgrade, but user settings of preferred CRI socket only persist in kubelet config file on disk. Kubeadm needs to store CRI socket configuration from disk to apply host-specific configuration and stop passing flag for CRI endpoint while assuming fields in config are set.

1. init: upload a global kubelet configuration with cri socket
2. join: can override it using --config
3. upgrade: re-download global one, but should use local kubelet configuration firstly

In one word, no annotation is needed, and it is replaced by a configuration in kubelet.

[pacoxu]

/sig cluster-lifecycle
/area kubeadm

[k8s-ci-robot]

@pacoxu: The label(s) area/kubeadm cannot be applied, because the repository doesn't have them.

In response to this:

/sig cluster-lifecycle
/area kubeadm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

Kubeadm needs to store CRI socket configuration from disk to apply host-specific configuration and stop passing flag for CRI endpoint while assuming fields in config are set.

one question will be where / how do we persist this host specific config.

could be just on upgrade we read the already used value from config on disc in memory. download global config and patch the value from memory before writing to disk.

[pacoxu]

In the past, we save KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=k8s.m.daocloud.io/pause:3.9" in /var/lib/kubelet/kubeadm-flags.env.

Should we add a /var/lib/kubelet/kubeadm-config.yaml which is a node specific configurations for kubelet? We can do a merge/overwrite using it with the re-downloaded global configuration during kubeadm upgrade.

[neolit123]

Should we add a /var/lib/kubelet/kubeadm-config.yaml which is a node specific configurations for kubelet?

sounds like an interesting idea. would like to see what others think too.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vaibhav2107]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

@pacoxu should we freeze this one?
i would assume you don't have bandwidth for it.

[pacoxu]

@pacoxu should we freeze this one?
i would assume you don't have bandwidth for it.

And the upgrade plan for this is not clear and may introduce more complexity/work(than I thought it was) on kubeadm.

/close

[k8s-ci-robot]

@pacoxu: Closing this issue.

In response to this:

@pacoxu should we freeze this one?
i would assume you don't have bandwidth for it.

And the upgrade plan for this is not clear and may introduce more complexity/work on kubeadm.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

And the upgrade plan for this is not clear and may introduce more complexity/work on kubeadm.

i think it's in the right direction, though.
the Node annotation is a bit of a hack.

[pacoxu]

And the upgrade plan for this is not clear and may introduce more complexity/work on kubeadm.

i think it's in the right direction, though. the Node annotation is a bit of a hack.

Yes, It is still a valid change.