Migration from k3os to openSUSE MicroOS

[mysticaltech]

Recently Rancher, the creators of k3s and k3os has been bought by SUSE. And in doing so, they've dropped official support for k3os (k3s on the other hand is thriving and has been separated from Rancher).

I went on to contact Jacob Blain Christen, the lead maintainer of k3os, and he told me that he'll continue to do releases on the weekends and that the project could live on if the community maintained it.

However, that is not a stable backing for this project, so I made my own research and concluded that OpenSuse MicroOS, has HUGE backing has it piggybacks on Tumbleweed, a major OpenSuse distro, and has stable and automated transactional updates, as such it's now the best OS to replace k3os.

[mysticaltech]

@sysrich I saw your article here https://rootco.de/2020-12-09-microos-pi-network-monitor/. I know it does not explain how to run MicroOS on Hetzner, but it says that Hetzner has included the ISO, unfortunately running hcloud iso list does not show it anymore. I am left with thinking about doing something like this in rescue mode:

export MICROOS_DISK=" https://downloadcontent.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-ContainerHost-SelfInstall.raw.xz"
curl -sL $MICROOS_DISK | xz -d | dd of=/dev/sda status=progress

• I will try it out, but wanted to get your opinion if possible, do you think it will work?
• After that, I just plan to issue a shutdown -r now command to reboot. Do you think networking will work OOTB?
• And does MicroOS support Hetzner user-data?
• If not, I can mount the disk just after dd finishes, but where do embed the ignition or combustion file? Basically, I want to install my SSH public key and k3s, or at least the former, if I can install k3s via SSH later on.

Last but not least, if you have done this before or have any user-data example or general guidance, it would be very much appreciated.

Am trying you move away from k3os as soon as possible, and I know that MicroOS is a much better solution (thanks to your hard work in part I would imagine)! 🙏

[spigell]

Watching for your progress, thanks for sharing! I also stuck with k3os and my bare metal setup. Seeking for similar functionality from another os.

[mnencia]

About installing MicroOS on Hetzner, did you see https://github.com/balta3/hetzner-microos? It could give some useful hints in the direction of an automated install ok MicroOS on hetzner.cloud

[mysticaltech]

Thanks, @mnencia, yes, thanks for sharing, I saw it too. It is interesting. But that method seems simpler https://major.io/2021/08/20/deploy-fedora-coreos-in-hetzner-cloud/.

I believe OpenSuse MicroOS inspired itself a good deal from Fedora CoreOS, as it even supports "ignition", maybe it even shares some code, so that method should work! 🤞

Will keep you folks posted, ASAP. Hopefully this week.

[mysticaltech]

Folks, recap on the advances. Both methods above have failed.

• The one with dd created a read-only filesystem, even for '/boot/writable' that is not supposed to be read-only. Probably because of the sensitivity of btrfs. So we cannot add the ignition file for SSH.
• The one from hetzner-microos, used kexec to launch the tumbleweed yast installer, all manual, not what we want, and does not give a proper MicroOS, just Tumbleweed with transactional updates if you choose so during the setup. So no go!

However, the good news is that MicroOS has images already loaded with k3s, so we have this option https://downloadcontent.opensuse.org/tumbleweed/appliances/iso/openSUSE-MicroOS.x86_64-k3s-SelfInstall.iso

For now, attempts to install it via kexec on the rescue env have failed, because of space limitations. Even when I load it on volumes. But there are still things to try and I have already requested Hetzner support to add it to their list of ISO (as a plan B), hopefully, they'll do so.

Let's see! 🤞 If you can think of something, do not hesitate!

[mnencia]

• The one with dd created a read-only filesystem, even for '/boot/writable' that is not supposed to be read-only. Probably because of the sensitivity of btrfs. So we cannot add the ignition file for SSH.

I think that's by purpose. You can mark the filesystem as rw by running

btrfs property set /mnt ro false

[mysticaltech]

Oh wow, will try ASAP @mnencia! 🙏

[mnencia]

I tried it, but Ignition does not read the configuration, probably because the partition is not labeled "ignition".

What about trying to use cloud-init? It should not require modifying the image, but only defining the right user-data content.

[mysticaltech]

Awesome! Did you create a folder in /boot/writable called ignition, and copy your generated config.ign to it? See https://en.opensuse.org/Portal:MicroOS/Ignition and https://en.opensuse.org/Portal:MicroOS/Design.

That's the format of mine for instance:
{"ignition":{"version":"3.0.0"},"passwd":{"users":[{"name":"root","sshAuthorizedKeys":["ssh-ed25519 ___ me@email.com"]}]}}

Also, with the Hetzner console, you can basically "plug a virtual monitor in the server' and see the boot process, which will tell you if the ignition is picked up or not, or what are the errors generated. If it goes too fast, just use screen recording software.

[mysticaltech]

Also, if you are going to retry, do not forget to resize the btrfs after dd, so it takes the whole space left on the disk.

Last but not least, this image is probably best, because it has k3s installed in it already, we just need to see how it's done and how it updates, with transactional updates too or not. If it uses transactional updates then great, if not we can just install the k3s tumbleweed package in the vanilla MicroOS (above), and have it update in a transactional manner along with other OS updates.

https://mirrorcache.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-16.0.0-k3s-SelfInstall-Snapshot20220201.raw.xz

Source: https://mirrorcache.opensuse.org/tumbleweed/appliances/

[mysticaltech]

@mnencia About cloud-init, it could work and is worth a shot too. My initial attempts failed, but I did not try everything.

About ignition, another thing worth a shot is creating ignition/config.ign on the root of an attached volume!

[mnencia]

Creating ignition/config.ign in the root doesn't work. Let's try with /boot/writable/ignition/config.ign

[mysticaltech]

The docs are not great, but https://en.opensuse.org/Portal:MicroOS/Design reveals the proper ignition location! The above should work 🤞

[mnencia]

I tried putting it under /boot/writable, mounted with the command mount -o subvol=/@/boot/writable /dev/sda4 /mnt, but it doesn't work anyway. Probably there is some subtle requirement that is missing.

[sysrich]

As documented here: https://en.opensuse.org/Portal:MicroOS/Ignition The only valid location for an ignition configuration is a seperate storage device with a volume name of ignition and a subdirectory named ignition

…

On Fri 4. Feb 2022 at 11:43, Marco Nenciarini ***@***.***> wrote: I tried putting it under /boot/writable, mounted with the command mount -o subvol=/@/boot/writable /dev/sda4 /mnt, but it doesn't work anyway. Probably there is some subtle requirement that is missing. — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJENIDMI73FRHOO7SZSPXLUZOUT7ANCNFSM5NFVEL2A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mysticaltech]

Thank you @sysrich, really appreciate the guidance! 🙏 I obviously had completely missed that! 🤦

Will attempt it with an attached volume. @mnencia If you are going to try this before me, just wanted to share this new finding of mine, the hcloud cli has an add-label command that could be handy here (if you didn't know already).

[phaer]

If it's possible to resize the btrfs filesystem ourselves from hetzners rescue system, we could maybe just add a small, labeled partition at the end? That one wouldn't require an external volume. Going to test that :)

[mnencia]

I think the add-label option in hcloud is about Hetzner metadata. However, Ignition doesn't run even creating a volume like the following

/dev/sdb1: LABEL="ignition" UUID="3dc811c0-9891-4e0f-8b94-1cc38f3967aa" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="ignition" PARTUUID="8fbab7ef-6be0-46b6-b594-53d9847a9e8d"

Looking in the system and initrd content it looks like there is no ignition handling code running in the SelfInstall flavor.

[mnencia]

I've tried with the volume and MicroOS based on Leap 15.3 (Experimental) and it works

Image: https://download.opensuse.org/repositories/openSUSE:/Leap:/Micro:/5.1/images/openSUSE-Leap-Micro.x86_64-Default.raw.xz

[mysticaltech]

@mnencia Great find, and well done. If we can find the Default raw image for MicroOS based on Tumbleweed that works, that's it!

Maybe the ignition logic for these is located somewhere else? And does not support volumes.

@phaer Please let us know if you try with partioniong and it works with the k3s self install image, it would be ideal!

Just FYI folks, when I ran dd myself yesterday, I had to run parted -l to correct a size mismatched error that was showing up on fdisk -l /dev/sda.

Also, I believe it's important to rezize the btrfs to max. Saying all this because it could perhaps influence the ignition seeking mechanisms on Tumbleweed based images?!

[phaer]

Hi, my approach did not work with the k3s self install image, but it did work with the one based on leap 15.3, as recommended by @mnencia.

I did not yet succeed to install k3s on this, as the default repo seems unavailable after boot.

Here's the terraform snippet I used to provision my test server. /root/config.ignwas provisioned beforehand via poseidon/ct provider:

  provisioner "remote-exec" {
    inline = [
      "set -ex",
      "gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0x22C07BA534178CD02EFE22AAB88B2FD43DBDC284",
      "export MIRROR_URL='https://download.opensuse.org/repositories/openSUSE:/Leap:/Micro:/5.1/images'",
      "export IMAGE_NAME='openSUSE-Leap-Micro.x86_64-Default.raw.xz'",
      "export IMAGE_URL=$MIRROR_URL/$IMAGE_NAME",
      "curl --progress-bar -L -o $IMAGE_NAME $IMAGE_URL",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256 $IMAGE_URL.sha256",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256.asc $IMAGE_URL.sha256.asc",
      "gpg --verify $IMAGE_NAME.sha256.asc",
      "sha256sum -c $IMAGE_NAME.sha256",
      "cat $IMAGE_NAME | xz -d | dd of=/dev/sda status=progress",
      "sgdisk -e /dev/sda",
      "partprobe /dev/sda",
      "parted -s /dev/sda resizepart 3 99%",
      "parted -s /dev/sda mkpart primary ext2 99% 100%",
      "mount /dev/sda3 /mnt/ && btrfs filesystem resize max /mnt && umount /mnt",
      "mke2fs -L ignition /dev/sda4",
      "mount /dev/sda4 /mnt",
      "mkdir /mnt/ignition",
      "cp /root/config.ign /mnt/ignition/config.ign",
      "umount /mnt",
      "shutdown -r +1",
      "sleep 1",
      "exit 0"
    ]
 

[mnencia]

This works and the resulting system is executing ignition

MICROOS_DISK="https://download.opensuse.org/tumbleweed/appliances/openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2"
curl -sL "${MICROOS_DISK}" -o /tmp/MicroOS.qcow2
qemu-img convert -p -f qcow2 -O host_device /tmp/MicroOS.qcow2 /dev/sda

[mysticaltech]

Wonderful! You made my day @mnencia! :-) Well done everyone. Now serious business can begin!

[mysticaltech]

Now the next steps are very clear:

• Check the partition size, resize to max if necessary (like @phaer did above)
• Verify networking, especially the private network interface, which needs to work well before anything else can happen. It should receive DHCP, and maybe we need to add routes to the Hetzner GW to it like done in the current code.
• Then, we have to run k3s for each of the three configurations, starting with the control plane.

I will definitely look at this as soon as I can but if you continue the work, please do not hesitate to share your success and failures, so we advance faster!

The last step will be to configure the cluster with kured for automatic safe rebooting after upgrades, and also transactional-updates, which luckily for us has full support for it.

[mnencia]

This list of commands works for me.

I had to remove the gpg and sha256sum verification because they do not match.

[
      "set -ex",
      "gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0x22C07BA534178CD02EFE22AAB88B2FD43DBDC284",
      "MIRROR_URL='https://mirrorcache.opensuse.org/tumbleweed/appliances/'",
      "IMAGE_NAME='openSUSE-MicroOS.x86_64-k3s-kvm-and-xen.qcow2'",
      "IMAGE_URL=$MIRROR_URL/$IMAGE_NAME",
      "curl --progress-bar -L -o $IMAGE_NAME $IMAGE_URL",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256 $IMAGE_URL.sha256",
      "curl --progress-bar -L -o $IMAGE_NAME.sha256.asc $IMAGE_URL.sha256.asc",
      # "gpg --verify $IMAGE_NAME.sha256.asc", # TODO: this doesn't match
      # "sha256sum -c $IMAGE_NAME.sha256", # TODO: this doesn't match
      "qemu-img convert -p -f qcow2 -O host_device $IMAGE_NAME /dev/sda",
      "sgdisk -e /dev/sda",
      "partprobe /dev/sda",
      "parted -s /dev/sda resizepart 4 99%",
      "parted -s /dev/sda mkpart primary ext2 99% 100%",
      "mount /dev/sda4 /mnt/ && btrfs filesystem resize max /mnt && umount /mnt",
      "mke2fs -L ignition /dev/sda5",
      "mount /dev/sda5 /mnt",
      "mkdir /mnt/ignition",
      "cp /root/config.ign /mnt/ignition/config.ign",
      "umount /mnt",
      "shutdown -r +1",
      "sleep 1",
      "exit 0"
]

[mysticaltech]

Beautiful! Yes, image verification is not a big issue for now.

[phaer]

I had to remove the gpg and sha256sum verification because they do not match.

Interesting, I had the same problem when I tried that image, but just failed to reproduce it. Could be that those keys were just updated, or maybe there's inconsistent state between mirrors and we got different redirects.

@mnencia Could you try if verification works if you use the following mirror for all 3 files (qcow, sha256sum, gpg signature)? If so, I would report that issue upstream.

MIRROR_URL='http://mirror.easyname.at/opensuse/tumbleweed/appliances/'

It is of course most important to get it to work at all, so great that it works without verification as a first step! 🎉

[mysticaltech]

More good news team, someone was kind enough to reply and explain how to do this! 🍾

[mysticaltech]

So this is the proper package to work with: https://build.opensuse.org/package/show/devel:kubic/k3s

[mysticaltech]

Was able to execute some of the commands, it re-packaged v1.23.3+k3s1, but the test build failed on my machine.

Some weird errors, about failed DNS requests to proxy.golang.com (see screenshot). Tried to disable the proxy, was not successful yet. I suspect, that my environment itself must not be compatible, as I am on Fedora, and ideally, that build process is tuned for Tumbleweed machines. So maybe will try from a live openSUSE USB at some point.

Anyways, there is a path forward to try to speed up releases, now we know what the flow is, and how to submit upgrade requests ourselves.

[mysticaltech]

About Node upgrades, one came through, and the /var/run/reboot-required showed up as planned. Also, kured uncordoned a few nodes, but before, I believe an error happens, because it pod itself restarts, but not the node.

Have replaced the image with the one from openSUSE as initially suggested by @mnencia, this one registry.opensuse.org/kubic/kured:1.9.1, and will report back tomorrow if the node rebooted or not.

[mysticaltech]

Something else we need to fix is SSH hardening. It seems that password auth is still active. And the ports that are hit, do not make sense to me, as those are supposed to be blocked by the firewall, just checked.

[mnencia]

This is what happened on my control node:

time="2022-02-09T07:19:03Z" level=info msg="Reboot required"
time="2022-02-09T07:19:03Z" level=info msg="Acquired reboot lock"
time="2022-02-09T07:19:03Z" level=info msg="Draining node k3s-control-plane-0"
WARNING: ignoring DaemonSet-managed Pods: kube-system/hcloud-csi-node-6xvnz, kube-system/kured-8wklm
evicting pod kube-system/hcloud-cloud-controller-manager-fcc9fb55c-spqqt
time="2022-02-09T07:19:04Z" level=info msg="Running command: [/usr/bin/nsenter -m/proc/1/ns/mnt -- /usr/bin/systemctl reboot] for node: k3s-control-plane-0"
time="2022-02-09T07:19:04Z" level=warning msg="nsenter: can't execute '/usr/bin/systemctl reboot': No such file or directory" cmd=/usr/bin/nsenter std=err
time="2022-02-09T07:19:04Z" level=fatal msg="Error invoking reboot command: exit status 127"

It looks like it is trying to execute /usr/bin/systemctl reboot as a file.

That's weird as https://github.com/weaveworks/kured/blob/main/cmd/kured/main.go#L666 should correctly split it in subparts.

[mnencia]

The nsenter command seems to work when executed from inside the pod

/ # /usr/bin/nsenter -m/proc/1/ns/mnt -- /usr/bin/systemctl is-system-running
running

[mnencia]

If you put quote around the command you get a similar error

/ # /usr/bin/nsenter -m/proc/1/ns/mnt -- '/usr/bin/systemctl is-system-running'
nsenter: can't execute '/usr/bin/systemctl is-system-running': No such file or directory

[phaer]

Can't contribute anything to the reboot command atm, but

Something else we need to fix is SSH hardening. It seems that password auth is still active. And the ports that are hit, do not make sense to me, as those are supposed to be blocked by the firewall, just checked.

Disabling password auth would definitely be a good idea. But those ports seem harmless. The logged ones are source ports, so the first log line means that 221.131.165.75 is connecting from port 56702 to our local port 22 (implicitly, because thats the only one our sshd listens to.

[mysticaltech]

@mnencia Good news, the registry.opensuse.org/kubic/kured:1.9.1 worked in rebooting my node, because we do not have to enter the reboot command manually, which apparently seems tricky! Should have listened to you since the beginning :)

[mysticaltech]

Just pushed created a new kured.yaml from the file you dumped on Kubic @phaer, so that apparently works in rebooting the nodes. You can pull it on staging.

[mysticaltech]

However, it rendered, my control-plane unaccessible. So I guess, Kured in more for worker nodes, also saw that in the docs yesterday. I will comb the k3s logs for more details.

[mysticaltech]

@phaer About the SSH ports, thanks for explaining! Definitely we can remove password auth through ignition normally.

[mysticaltech]

FYI, when my first_control_plane stopped "correctly" it seems, to reboot. Accessing these logs through journalctl -u k3s-server | less if you know a better way, please let me know.

[mysticaltech]

And then, there is this error again and again! And also tried to re-run k3s-sever without the cluster-init:true param in the config, but to no avail, the same thing.

[mysticaltech]

My guess is that we first ban control-plane nodes from kured, make sure it works correctly on the workers, and then, see what can be done with the former.

[mysticaltech]

If we exhaust all avenues, maybe we could go back to https://github.com/rancher/system-upgrade-controller, like used in k3os!

[mnencia]

I found the initial issue with the restart command. Once you say it you cannot believe you missed it until then:

diff --git a/kured/patch.yaml b/kured/patch.yaml
index bfec414..bf72a0c 100644
--- a/kured/patch.yaml
+++ b/kured/patch.yaml
@@ -17,4 +17,4 @@ spec:
         - name: kured
           command:
             - /usr/bin/kured
-            - --reboot-command="/usr/bin/systemctl reboot"
+            - --reboot-command=/usr/bin/systemctl reboot

In pr #51

[mnencia]

Regarding the restart of control-plane nodes, the main issue here is the fact that there are only two nodes. Etcd does not work well with only two nodes because if one restart the other loses the quorum anyway, so the cluster is unusable until the restarted node works again. There are also other issues:

• The kubeconfig.yaml file only points to k3s-control-plane-0, so if for any reason that node dies, you have to repoint it to the second node.
• In terraform k3s-control-plane-0 node is special, if you need to reinstall the node (for example because of a disk corruption) it is really difficult to replace it.

[mnencia]

I have a cluster with 3 masters, let's see how it reacts to the restart.

[mysticaltech]

Perfect! Thanks for the explanations.

[phaer]

Thanks for all that work on upgrades!

Shall we consider to split the different topics in this thread (k3s packaging, ssh hardening and reboot-on-upgrades atm) into different issues? We could also use GitHubs project feature to keep discussion a bit more browseable.

Another question I'd like to discuss is: Are there still benefits in using k3s? As we are using MicroOS, kubic with kubeadm seems to be comparable in setup complexity and according to k3s README the differences don't seem to big these days?

[mysticaltech]

Great suggestions @phaer! Was thinking along the same lines. Probably we need to close this issue and move on to other ones, as it's +100 comments long 😅 About projects on Github, I've never used those, but would be very curious to try, let me know what you think both.

About Kubic, it's heavyweight because of kubeadm, and even the Kubic team seems to be exploring k3s, as a lightweight Kubernetes option, see https://build.opensuse.org/package/show/devel:kubic/k3s.

Last but not least, as you know, the Rancher folks just joined SUSE and they are the core maintainers of k3s, especially Darren Sheppard, the "inventor" of k3s, who is now SUSE's Chief Architect!

And yesterday have managed to inform him about our woes with the infrequent k3s version upgrades released via the "openSUSE Factory" channel. See here.

So Kubic with kubeadm is definitely a plan B, but perhaps let's just give k3s another chance. As the upgrades, for instance, are a lot simpler! Just replace the binary... For kubeadm, oh my!

[mysticaltech]

Folks, we have successfully moved our staging branch to openSUSE MicroOS, now we are fixing bugs and getting the flow right. Please open other issues for that.

Thanks again @mnencia and @phaer, this was amazing teamwork! :)

[mysticaltech]

This was just merged into master!