Failure to upgrade a flatcar cluster due to patch "Refactor bootstrap-os (#10983)" #11268
Labels
kind/bug
Categorizes issue or PR as related to a bug.
lifecycle/stale
Denotes an issue or PR has remained open with no activity and has become stale.
What happened?
When upgrading a 5 node flatcar cluster using Kubespray v2.25.0 the following task fails (on all nodes, only copied the first error message).
The cluster is currently in kubernetes v1.28.6 version deployed by Kubespray v2.24.1.
I suspect the patch which is causing my issues is Refactor bootstrap-os (#10983) especially the changes to the
roles/bootstrap-os/tasks/flatcar.yml
file. But I'm not sure how to workaround it. It looks like it comes from an ansible change to make ansible work out of the box with Flatcar, but in my case it's not working.What did you expect to happen?
Pre-tests to pass and smooth upgrade. I've followed this same repetitive process for versions from 2.27.x to 2.28.x
How can we reproduce it (as minimally and precisely as possible)?
The marvinpac-it/mvp-cluster repository contains my inventory. Beyond activating nginx, the only change I made for deploying the cluster on flatcar was to set the
bin_dir
variable to/opt/bin
. This is as I understand a pre-requisite for flatcar knowing that the /usr folder is read-only.OS
Ansible server:
Linux 5.15.0-107-generic x86_64
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Kubernetes nodes:
Linux 6.1.90-flatcar x86_64
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3815.2.3
VERSION_ID=3815.2.3
BUILD_ID=2024-05-21-1124
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3815.2.3 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3815.2.3:::::::*"
Version of Ansible
ansible [core 2.16.7]
config file = /tmp/kkk/kubespray/ansible.cfg
configured module search path = ['/tmp/kkk/kubespray/library']
ansible python module location = /tmp/kkk/kubespray/venv/lib/python3.10/site-packages/ansible
ansible collection location = /home/olivier/.ansible/collections:/usr/share/ansible/collections
executable location = /tmp/kkk/kubespray/venv/bin/ansible
python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/tmp/kkk/kubespray/venv/bin/python3)
jinja version = 3.1.4
libyaml = True
Version of Python
Python 3.10.12
Version of Kubespray (commit)
7e0a407
Network plugin used
calico
Full inventory with variables
Can't make this command work, but inventory is fully described here: https://github.com/marvinpac-it/mvp-cluster
Command used to invoke ansible
ansible-playbook -i inventory/mvp-cluster/hosts.yaml -u core --key-file ~/.ssh/flatcar_ssh.pem -b -e kube_version=v1.29.5 upgrade-cluster.yml
Output of ansible run
https://gist.github.com/oboudry-mvp/1b1271d0880f1cd80f07610e22463f4e
Anything else we need to know
I tried to add the following environment variable that I found in a post:
-e '{"ansible_interpreter_python_fallback":["/opt/bin/pypy3/bin/python"]}'
It goes a bit further, but at some point it fails too. Output below:
https://gist.github.com/oboudry-mvp/8f345c6fcba1c178883ec6f98fb192ce
The text was updated successfully, but these errors were encountered: