Initial setup of a k8s cluster with kubespray breaks if kube-vip, metallb is enabled and cillium with kube-proxy replacment

[samfili]

What happened?

Running an initial cluster creation breaks always on registering first master if kube-vip, metallb is enabled and cillium with kube-proxy replacment

What did you expect to happen?

Initialization of the first master failed.

How can we reproduce it (as minimally and precisely as possible)?

Deploy a minimal cluster in a fresh environment and activate kube-vip, metallb and cillium with kube-proxy replacment beforehand via addons.yml.

OS

Linux 6.8.0-36-lowlatency x86_64
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Version of Ansible

ansible [core 2.16.7]
config file = /Users/rexic/Docs/testkube_dev/kubespray/kubespray-install/kubespray-2.25.0/ansible.cfg
configured module search path = ['/Users/rexic/Docs/testkube_dev/kubespray/kubespray-install/kubespray-2.25.0/library']
ansible python module location = /Users/rexic/Docs/testkube_dev/kubespray/kubespray-install/.venv/lib/python3.12/site-packages/ansible
ansible collection location = /Users/sam/.ansible/collections:/usr/share/ansible/collections
executable location = /Users/rexic/Docs/testkube_dev/kubespray/kubespray-install/.venv/bin/ansible
python version = 3.12.4 (main, Jun 6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] (/Users/rexic/Docs/testkube_dev/kubespray/kubespray-install/.venv/bin/python)
jinja version = 3.1.4
libyaml = True

Version of Python

Python 3.12.4

Version of Kubespray (commit)

Collection (2.25.0)

Network plugin used

cilium

Full inventory with variables

[all]
k8s-dev-master01.testkube.local ansible_host=10.10.10.16 ip=10.10.10.16 etcd_member_name=etcd1
k8s-dev-worker01.testkube.local ansible_host=10.10.10.23 ip=10.10.10.23
k8s-dev-worker02.testkube.local ansible_host=10.10.10.24 ip=10.10.10.24
k8s-dev-worker03.testkube.local ansible_host=10.10.10.25 ip=10.10.10.25

[kube_control_plane]
k8s-dev-master01.testkube.local

[etcd]
k8s-dev-master01.testkube.local

[kube_node]
k8s-dev-worker01.testkube.local
k8s-dev-worker02.testkube.local
k8s-dev-worker03.testkube.local

[calico_rr]

[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

Command used to invoke ansible

ansible-playbook -i inventory/testkube-dev/inventory.ini -b -u ubuntu -v --private-key=~/.ssh/k8s_id_rsa cluster.yml

Output of ansible run

TASK [kubernetes/control-plane : Kubeadm | Initialize first master]
fatal: [k8s-dev-master01.testkube.local]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["timeout", "-k", "300s", "300s", "/usr/local/bin/kubeadm", "init", "--config=/etc/kubernetes/kubeadm-config.yaml", "--ignore-preflight-errors=all", "--skip-phases=addon/coredns,addon/kube-proxy", "--upload-certs"], "delta": "0:05:00.019405", "end": "2024-06-30 17:29:43.438077", "failed_when_result": true, "msg": "non-zero return code", "rc": 124, "start": "2024-06-30 17:24:43.418672", "stderr": "W0630 17:24:43.484417 10215 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.234.0.10]; the provided value is: [169.254.25.10]\n\t[WARNING Port-6443]: Port 6443 is in use\n\t[WARNING Port-10259]: Port 10259 is in use\n\t[WARNING Port-10257]: Port 10257 is in use\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\n\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists\n\t[WARNING Port-10250]: Port 10250 is in use", "stderr_lines": ["W0630 17:24:43.484417 10215 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.234.0.10]; the provided value is: [169.254.25.10]", "\t[WARNING Port-6443]: Port 6443 is in use", "\t[WARNING Port-10259]: Port 10259 is in use", "\t[WARNING Port-10257]: Port 10257 is in use", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists", "\t[WARNING FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists", "\t[WARNING Port-10250]: Port 10250 is in use"], "stdout": "[init] Using Kubernetes version: v1.29.5\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[certs] Using certificateDir folder "/etc/kubernetes/ssl"\n[certs] Using existing ca certificate authority\n[certs] Using existing apiserver certificate and key on disk\n[certs] Using existing apiserver-kubelet-client certificate and key on disk\n[certs] Using existing front-proxy-ca certificate authority\n[certs] Using existing front-proxy-client certificate and key on disk\n[certs] External etcd mode: Skipping etcd/ca certificate authority generation\n[certs] External etcd mode: Skipping etcd/server certificate generation\n[certs] External etcd mode: Skipping etcd/peer certificate generation\n[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation\n[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation\n[certs] Using the existing "sa" key\n[kubeconfig] Using kubeconfig folder "/etc/kubernetes"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/super-admin.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"\n[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"\n[control-plane] Using manifest folder "/etc/kubernetes/manifests"\n[control-plane] Creating static Pod manifest for "kube-apiserver"\n[control-plane] Creating static Pod manifest for "kube-controller-manager"\n[control-plane] Creating static Pod manifest for "kube-scheduler"\n[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"\n[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"\n[kubelet-start] Starting the kubelet\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 5m0s\n[kubelet-check] Initial timeout of 40s passed.", "stdout_lines": ["[init] Using Kubernetes version: v1.29.5", "[preflight] Running pre-flight checks", "[preflight] Pulling images required for setting up a Kubernetes cluster", "[preflight] This might take a minute or two, depending on the speed of your internet connection", "[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'", "[certs] Using certificateDir folder "/etc/kubernetes/ssl"", "[certs] Using existing ca certificate authority", "[certs] Using existing apiserver certificate and key on disk", "[certs] Using existing apiserver-kubelet-client certificate and key on disk", "[certs] Using existing front-proxy-ca certificate authority", "[certs] Using existing front-proxy-client certificate and key on disk", "[certs] External etcd mode: Skipping etcd/ca certificate authority generation", "[certs] External etcd mode: Skipping etcd/server certificate generation", "[certs] External etcd mode: Skipping etcd/peer certificate generation", "[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation", "[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation", "[certs] Using the existing "sa" key", "[kubeconfig] Using kubeconfig folder "/etc/kubernetes"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/super-admin.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"", "[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"", "[control-plane] Using manifest folder "/etc/kubernetes/manifests"", "[control-plane] Creating static Pod manifest for "kube-apiserver"", "[control-plane] Creating static Pod manifest for "kube-controller-manager"", "[control-plane] Creating static Pod manifest for "kube-scheduler"", "[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"", "[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"", "[kubelet-start] Starting the kubelet", "[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 5m0s", "[kubelet-check] Initial timeout of 40s passed."]}

NO MORE HOSTS LEFT

PLAY RECAP
k8s-dev-master01.testkube.local : ok=586 changed=120 unreachable=0 failed=1 skipped=628 rescued=0 ignored=3
k8s-dev-worker01.testkube.local : ok=419 changed=89 unreachable=0 failed=0 skipped=454 rescued=0 ignored=1
k8s-dev-worker02.testkube.local : ok=419 changed=89 unreachable=0 failed=0 skipped=450 rescued=0 ignored=1
k8s-dev-worker03.testkube.local : ok=419 changed=89 unreachable=0 failed=0 skipped=450 rescued=0 ignored=1

Sunday 30 June 2024 17:29:43 +0300 (0:20:26.892) 0:43:19.769

kubernetes/control-plane : Kubeadm | Initialize first master ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1226.89s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 91.76s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 90.40s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43.59s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 42.35s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 41.43s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 33.83s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 23.72s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 22.03s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20.11s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17.40s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17.20s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15.62s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15.31s
container-engine/skopeo : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14.77s
container-engine/crictl : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14.20s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14.12s
download : Download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.59s
container-engine/cri-o : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12.61s
download : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12.32s

Anything else we need to know

No response

[Etrenak]

I have the same issue when kube_vip is enabled and the cluster.yml playbook is used to create the cluster. The workaround is to create the cluster without kube_vip and then run cluster.yml again, with kube_vip enabled.