immucore fails to create directories define in bind_mounts

[jimmykarily]

Installing kairos with this config (as part of #1114):

#cloud-config

install:
  device: "auto"
  auto: true
  reboot: true
  bind_mounts:
  - /etc/coco
  - /etc/containerd

hostname: debugging-station-{{ trunc 4 .MachineID }}

users:
- name: kairos
  passwd: kairos

k3s:
  enabled: true

# Specify the bundle to use
bundles:
- targets:
  - run://quay.io/kairos/community-bundles:system-upgrade-controller_latest
  - run://quay.io/kairos/community-bundles:cert-manager_latest
  - run://quay.io/kairos/community-bundles:kairos_latest
  - run://ttl.sh/kairos-testing/enclave-cc:8h

stages:
  before-install:
    - commands:
      - mkdir -p /etc/coco
      - mkdir -p /etc/containerd

kairos:
  entangle:
    enable: true

write_files:
- path: /var/lib/rancher/k3s/server/manifests/expose-ssh.yaml
  permissions: "0644"
  owner: "root"
  content: |
      apiVersion: v1
      kind: Secret
      metadata:
        name: ssh-entanglement
        namespace: kube-system
      type: Opaque
      stringData:
        network_token: <removed>
      ---
      apiVersion: entangle.kairos.io/v1alpha1
      kind: Entanglement
      metadata:
        name: ssh-entanglement
        namespace: kube-system
      spec:
         serviceUUID: "ssh"
         secretRef: "ssh-entanglement"
         host: "127.0.0.1"
         port: "22"
         hostNetwork: true

the 2 bind_mount directories are not created. The logs from /run/immucore/immucore.log:

<nil> INF Immucore commit=none compiled with=go1.20.2 version=v0.0.1
<nil> DBG cmdline content="BOOT_IMAGE=(loop0)/boot/vmlinuz console=tty1 console=ttyS0 root=LABEL=COS_ACTIVE net.ifnames=1 cos-img/filename=/cOS/active.img rd.emergency=reboot rd.shell=0 panic=5 security=selinux rd.cos.oemlabel=COS_OEM selinux=1 fsck.mode=force fsck.repair=yes systemd.crash_reboot=yes rd.emergency=reboot rd.shell=0 panic=5 systemd.crash_reboot systemd.crash_shell=0\n"
<nil> DBG Target device what=/cOS/active.img
<nil> DBG Target label what=/dev/disk/by-label/COS_ACTIVE
<nil> INF Booting on active/passive/recovery.
<nil> DBG Get state label what=/dev/disk/by-label/COS_STATE
<nil> DBG Get state label what=/dev/disk/by-label/COS_STATE
<nil> DBG Partition FS type type=ext4 what=/dev/disk/by-label/COS_STATE
<nil> DBG Partition FS type type=ext4 what=/dev/disk/by-label/COS_OEM
<nil> INF 1.
 <init> (background: false) (weak: false) (run: false)
2.
 <create-sentinel> (background: false) (weak: false) (run: false)
 <mount-tmpfs> (background: false) (weak: false) (run: false)
 <mount-state> (background: false) (weak: false) (run: false)
3.
 <discover-state> (background: false) (weak: false) (run: false)
4.
 <mount-root> (background: false) (weak: false) (run: false)
5.
 <mount-oem> (background: false) (weak: false) (run: false)
6.
 <rootfs-hook> (background: false) (weak: false) (run: false)
7.
 <load-config> (background: false) (weak: false) (run: false)
8.
 <mount-base-overlay> (background: false) (weak: false) (run: false)
 <custom-mount> (background: false) (weak: false) (run: false)
9.
 <mount-bind> (background: false) (weak: false) (run: false)
 <overlay-mount> (background: false) (weak: false) (run: false)
10.
 <write-fstab> (background: false) (weak: false) (run: false)
11.
 <initramfs-hook> (background: false) (weak: false) (run: false)

<nil> INF Setting sentinel file to=active_mode
<nil> INF mount done options=["rw"] type=tmpfs what=tmpfs where=/tmp
<nil> DBG fsck command cmd="fsck /dev/disk/by-label/COS_STATE -f -y"
<nil> INF mount done options=["ro"] type=ext4 what=/dev/disk/by-label/COS_STATE where=/sysroot/run/initramfs/cos-state
<nil> DBG fsck command cmd="fsck /sysroot/run/initramfs/cos-state/cOS/active.img -f -y"
<nil> DBG fsck error="exit status 8" out="fsck from util-linux 2.37.2\ne2fsck 1.46.4 (18-Aug-2021)\nfsck.ext2: Read-only file system while trying to open /sysroot/run/initramfs/cos-state/cOS/active.img\nDisk write-protected; use the -n option to do a read-only\ncheck of the device.\n" what=/sysroot/run/initramfs/cos-state/cOS/active.img
<nil> DBG udevadm trigger output=
<nil> DBG mount done TargetDevice=/dev/disk/by-label/COS_ACTIVE path=/sysroot targetImage=/cOS/active.img
<nil> DBG fsck command cmd="fsck /dev/disk/by-label/COS_ACTIVE -f -y"
<nil> DBG fsck error="exit status 8" out="fsck from util-linux 2.37.2\ne2fsck 1.46.4 (18-Aug-2021)\nWarning!  /dev/loop0 is mounted.\nfsck.ext2: Operation not permitted while trying to open /dev/loop0\nYou must have r/w access to the filesystem or be root\n" what=/dev/disk/by-label/COS_ACTIVE
<nil> INF mount done options=["ro","suid","dev","exec","async"] type=ext4 what=/dev/disk/by-label/COS_ACTIVE where=/sysroot
<nil> DBG fsck command cmd="fsck /dev/disk/by-label/COS_OEM -f -y"
<nil> INF mount done options=["rw","suid","dev","exec","async"] type=ext4 what=/dev/disk/by-label/COS_OEM where=/sysroot/oem
<nil> INF Running rootfs stage
<nil> INF getpartbylabel label=COS_PERSISTENT part={"filesystem_label":"unknown","label":"bios","mount_point":"","name":"vda1","read_only":true,"size_bytes":1048576,"type":"unknown","uuid":"7d1dd3e7-ee4c-4eea-8c7f-e9ef537eb1a7"}
<nil> INF getpartbylabel label=COS_PERSISTENT part={"filesystem_label":"COS_OEM","label":"oem","mount_point":"","name":"vda2","read_only":true,"size_bytes":67108864,"type":"ext4","uuid":"24da0d01-1652-4d5f-81e1-dd32d733def5"}
<nil> INF getpartbylabel label=COS_PERSISTENT part={"filesystem_label":"COS_RECOVERY","label":"recovery","mount_point":"","name":"vda3","read_only":true,"size_bytes":8589934592,"type":"ext4","uuid":"787197ec-5cc7-4873-a340-f916ab0a9a22"}
<nil> INF getpartbylabel label=COS_PERSISTENT part={"filesystem_label":"COS_STATE","label":"state","mount_point":"","name":"vda4","read_only":true,"size_bytes":16106127360,"type":"ext4","uuid":"8e83031f-6618-4d04-8530-ffc19e8c3ade"}
<nil> INF getpartbylabel label=COS_PERSISTENT part={"filesystem_label":"COS_PERSISTENT","label":"persistent","mount_point":"","name":"vda5","read_only":true,"size_bytes":39658192896,"type":"ext4","uuid":"76b049da-84f6-4024-a5d9-68b2203b51cc"}
<nil> DBG Mounting custom mounts mounts={"/dev/disk/by-label/COS_OEM":"/oem","/dev/disk/by-label/COS_PERSISTENT":"/usr/local"}
<nil> DBG fsck command cmd="fsck /dev/disk/by-label/COS_OEM -f -y"
<nil> DBG fsck error="exit status 8" out="fsck from util-linux 2.37.2\ne2fsck 1.46.4 (18-Aug-2021)\n/dev/vda2 is mounted.\ne2fsck: Cannot continue, aborting.\n\n\n" what=/dev/disk/by-label/COS_OEM
<nil> INF mount done options=["ro"] type=ext4 what=/dev/disk/by-label/COS_OEM where=/sysroot/oem
<nil> DBG fsck command cmd="fsck /dev/disk/by-label/COS_PERSISTENT -f -y"
<nil> INF mount done options=["rw"] type=ext4 what=/dev/disk/by-label/COS_PERSISTENT where=/sysroot/usr/local
<nil> INF
<nil> DBG Mounting overlays dirs=["/var","/etc","/srv"]
<nil> DBG Mounting binds mounts=["/etc/coco","/etc/containerd","/etc/systemd","/etc/modprobe.d","/etc/rancher","/etc/sysconfig","/etc/runlevels","/etc/ssh","/etc/ssl/certs","/etc/iscsi","/etc/zfs","/etc/cni","/etc/kubernetes","/home","/opt","/root","/snap","/var/snap","/usr/libexec","/var/log","/var/lib/rancher","/var/lib/kubelet","/var/lib/snapd","/var/lib/wicked","/var/lib/longhorn","/var/lib/cni","/usr/share/pki/trust","/usr/share/pki/trust/anchors","/var/lib/ca-certificates"]
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-coco.bind where=/sysroot/etc/coco
<nil> ERR executing mount callback error="mkdir /sysroot/etc/coco: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/etc-coco.bind where=/sysroot/etc/coco
<nil> ERR error="mkdir /sysroot/etc/coco: read-only file system"
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-containerd.bind where=/sysroot/etc/containerd
<nil> ERR executing mount callback error="mkdir /sysroot/etc/containerd: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/etc-containerd.bind where=/sysroot/etc/containerd
<nil> ERR error="mkdir /sysroot/etc/containerd: read-only file system"
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-systemd.bind where=/sysroot/etc/systemd
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-modprobe.d.bind where=/sysroot/etc/modprobe.d
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-rancher.bind where=/sysroot/etc/rancher
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-sysconfig.bind where=/sysroot/etc/sysconfig
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-runlevels.bind where=/sysroot/etc/runlevels
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-ssh.bind where=/sysroot/etc/ssh
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-ssl-certs.bind where=/sysroot/etc/ssl/certs
<nil> ERR executing mount callback error="mkdir /sysroot/etc/ssl/certs: file exists" options=["bind"] type=overlay what=/sysroot/usr/local/.state/etc-ssl-certs.bind where=/sysroot/etc/ssl/certs
<nil> ERR error="mkdir /sysroot/etc/ssl/certs: file exists"
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-iscsi.bind where=/sysroot/etc/iscsi
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-zfs.bind where=/sysroot/etc/zfs
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-cni.bind where=/sysroot/etc/cni
<nil> DBG Bind mount what=/sysroot/usr/local/.state/etc-kubernetes.bind where=/sysroot/etc/kubernetes
<nil> DBG Bind mount what=/sysroot/usr/local/.state/home.bind where=/sysroot/home
<nil> DBG Bind mount what=/sysroot/usr/local/.state/opt.bind where=/sysroot/opt
<nil> DBG Bind mount what=/sysroot/usr/local/.state/root.bind where=/sysroot/root
<nil> DBG Bind mount what=/sysroot/usr/local/.state/snap.bind where=/sysroot/snap
<nil> ERR executing mount callback error="mkdir /sysroot/snap: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/snap.bind where=/sysroot/snap
<nil> ERR error="mkdir /sysroot/snap: read-only file system"
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-snap.bind where=/sysroot/var/snap
<nil> DBG Bind mount what=/sysroot/usr/local/.state/usr-libexec.bind where=/sysroot/usr/libexec
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-log.bind where=/sysroot/var/log
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-rancher.bind where=/sysroot/var/lib/rancher
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-kubelet.bind where=/sysroot/var/lib/kubelet
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-snapd.bind where=/sysroot/var/lib/snapd
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-wicked.bind where=/sysroot/var/lib/wicked
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-longhorn.bind where=/sysroot/var/lib/longhorn
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-cni.bind where=/sysroot/var/lib/cni
<nil> DBG Bind mount what=/sysroot/usr/local/.state/usr-share-pki-trust.bind where=/sysroot/usr/share/pki/trust
<nil> DBG Bind mount what=/sysroot/usr/local/.state/usr-share-pki-trust-anchors.bind where=/sysroot/usr/share/pki/trust/anchors
<nil> DBG Bind mount what=/sysroot/usr/local/.state/var-lib-ca-certificates.bind where=/sysroot/var/lib/ca-certificates
<nil> ERR error="4 errors occurred:\n\t* mkdir /sysroot/etc/coco: read-only file system\n\t* mkdir /sysroot/etc/containerd: read-only file system\n\t* mkdir /sysroot/etc/ssl/certs: file exists\n\t* mkdir /sysroot/snap: read-only file system\n\n"
<nil> INF Running initramfs stage
<nil> DBG Source does not exists, not mounting in chroot what=/run/rootfsbase
<nil> DBG Source does not exists, not mounting in chroot what=/run/initramfs/live
<nil> DBG Closing chroot activeMounts=["/sysroot/sys","/sysroot/dev","/sysroot/dev/pts","/sysroot/dev/shm","/sysroot/proc","/sysroot/tmp","/sysroot/run"]
<nil> DBG Unmounting from chroot what=/sysroot/run
<nil> DBG Unmounting from chroot what=/sysroot/tmp
<nil> DBG Unmounting from chroot what=/sysroot/proc
<nil> DBG Unmounting from chroot what=/sysroot/dev/shm
<nil> DBG Unmounting from chroot what=/sysroot/dev/pts
<nil> DBG Unmounting from chroot what=/sysroot/dev
<nil> DBG Unmounting from chroot what=/sysroot/sys
<nil> INF 1.
 <init> (background: false) (weak: false) (run: false)
2.
 <mount-tmpfs> (background: false) (weak: false) (run: true)
 <create-sentinel> (background: false) (weak: false) (run: true)
 <mount-state> (background: false) (weak: false) (run: true)
3.
 <discover-state> (background: false) (weak: false) (run: true)
4.
 <mount-root> (background: false) (weak: false) (run: true)
5.
 <mount-oem> (background: false) (weak: false) (run: true)
6.
 <rootfs-hook> (background: false) (weak: false) (run: true)
7.
 <load-config> (background: false) (weak: false) (run: true)
8.
 <mount-base-overlay> (background: false) (weak: false) (run: true)
 <custom-mount> (background: false) (weak: false) (run: true)
9.
 <overlay-mount> (background: false) (weak: false) (run: true)
 <mount-bind> (error: 4 errors occurred:
	* mkdir /sysroot/etc/coco: read-only file system
	* mkdir /sysroot/etc/containerd: read-only file system
	* mkdir /sysroot/etc/ssl/certs: file exists
	* mkdir /sysroot/snap: read-only file system

) (background: false) (weak: false) (run: true)
10.
 <write-fstab> (background: false) (weak: false) (run: true)
11.
 <initramfs-hook> (background: false) (weak: false) (run: true)

Kairos version:

A custom build with the sdk bumped:

-       github.com/kairos-io/kairos-sdk v0.0.1
+       github.com/kairos-io/kairos-sdk v0.0.2-0.20230317071609-7a148fe5bb90

so that I can use this: kairos-io/kairos-sdk@7a148fe

kairos@debugging-station:~> cat /etc/os-release 
NAME="kairos-opensuse-leap"
VERSION="v1.6.1-11-g20659ae-dirty-k3s"
ID="kairos"
ID_LIKE="kairos-opensuse-leap"
VERSION_ID="v1.6.1-11-g20659ae-dirty-k3s"
PRETTY_NAME="kairos-opensuse-leap v1.6.1-11-g20659ae-dirty-k3s"
ANSI_COLOR="0;32"
BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues/new/choose"
HOME_URL="https://github.com/kairos-io/provider-kairos"
IMAGE_REPO="quay.io/kairos/kairos-opensuse-leap"
IMAGE_LABEL="latest"
GITHUB_REPO="kairos-io/provider-kairos"
VARIANT="core"
FLAVOR="opensuse-leap"

CPU architecture, OS, and Version:

kairos@debugging-station:~> uname -r
5.14.21-150400.24.46-default

Describe the bug
the directories in bind_mounts should be created and mounted as persistent

To Reproduce
see above config

[Itxaka]

confirmed. Weird, I expected that it will create those dirs, as some of the default ones arent in the base image and /etc has to be RW at that point as its on the RW_PATHS....

[Itxaka]

overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/overlay/etc/.overlay/upper,workdir=/run/overlay/etc/.overlay/work)

Manually creating the dir works after boot.... so maybe its a timing issue? /etc is not RW yet at that point?? That makes no sense

kairos@kairos-f4zw:~> sudo mkdir /etc/caca
kairos@kairos-f4zw:~> ls -ltra /etc/ca
ls: cannot access '/etc/ca': No such file or directory
kairos@kairos-f4zw:~> ls -ltra /etc/ca
ca-certificates/ caca/            
kairos@kairos-f4zw:~> ls -ltra /etc/caca
total 0
drwxr-xr-x 1 root root 420 Mar 21 09:00 ..
drwxr-xr-x 2 root root  40 Mar 21 09:00 .
``

[Itxaka]

Some of the bind mounts that we specify are not in the rootfs so it cannot be that, for example the default /etc/rancher does not exist in the rootfs:

+image-rootfs | ls: cannot access '/etc/rancher': No such file or directory

but then on boot:

<nil> DBG luetbuild/go/src/github.com/kairos-io/immucore/pkg/mount/fs.go:83 > Bind mount what=/sysroot/usr/local/.state/etc-rancher.bind where=/sysroot/etc/rancher

So creating new dirs under a RW path should work. Adding a branch with extra debugging to immucore to track this.

[Itxaka]

ah shit, its a dependency issue.

Mount binds should depend on overlay-mount to start, as overlay-mount mounts the RW paths.

Patch incoming