🌱 Run enclave-cc on Kairos

[mudler]

https://github.com/confidential-containers/enclave-cc

Acceptance criteria

• We have a documentation page showing how to run confidential workloads on Kairos
• We have a bundle that sets up enclave-cc and the operator accordingly
• Have an e2e example to show confidential workloads running on Kairos

Useful docs links:

• https://kairos.io/docs/advanced/bundles/#create-bundles

[jimmykarily]

The installation instructions fail on this step: https://github.com/confidential-containers/operator/blob/main/docs/INSTALL.md#create-custom-resource-cr

I'm using a k3d cluster and when the cc-operator-pre-install-daemon-5b8qf pod start it fails with an error:

Copying containerd-for-cc artifacts onto host                                                                                                                                              Failed to get D-Bus connection: Operation not permitted   

I think it has to do with this script trying to run systemctl: https://github.com/confidential-containers/operator/blob/a0fbbf40ad0848aee6c9ed90fbf7d001e50396c4/install/pre-install-payload/scripts/container-engine-for-cc-deploy.sh#L55

the failing container is running with:

 50     securityContext:                                                            
 51       privileged: true                                                          
 52       runAsUser: 0  

so I don't see how this could be a permissions issue.

[jimmykarily]

The Pod assumes dbus is running on the host: https://github.com/confidential-containers/operator/blob/a0fbbf40ad0848aee6c9ed90fbf7d001e50396c4/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml#L43

which is not the case for the k3d container.

[mudler]

I'd suggest to try on a real cluster with kairos and k3s as assumes you have services running, you might hit several limitations along the way with k3d that might block you differently

[mudler]

From the script you are linking: https://github.com/confidential-containers/operator/blob/a0fbbf40ad0848aee6c9ed90fbf7d001e50396c4/install/pre-install-payload/scripts/container-engine-for-cc-deploy.sh#LL48C8-L48C36

I'm assuming we should also add that /opt/... path to the persistent path during installation https://kairos.io/docs/advanced/customizing/#bind-mounts

[jimmykarily]

I switched to a kairos cluster. This is the next error:

│ Copying containerd-for-cc artifacts onto host                                                                                                                                              │
│ Restarting containerd                                                                                                                                                                      │
│ Failed to restart containerd.service: Unit containerd.service not found. 

containerd based cluster is in the requirements

[mudler]

I switched to a kairos cluster. This is the next error:

│ Copying containerd-for-cc artifacts onto host                                                                                                                                              │
│ Restarting containerd                                                                                                                                                                      │
│ Failed to restart containerd.service: Unit containerd.service not found. 

containerd based cluster is in the requirements

K3s is containerd based: https://docs.k3s.io/advanced#using-docker-as-the-container-runtime

[mudler]

I think the issue is it tries to restart a service and gives for guaranteed there is one. maybe we can slightly adapt their script to work on k3s?

[jimmykarily]

After inspecting how the operator installs the needed binaries, we came up with this plan (cc @mudler ):

• The operator is replacing containerd in a strange way, assuming containerd is managed with systemd (which is not the case in k3s). For that reason, we will replace containerd ouside the cluster in a kairos bundle with one built from their fork: https://github.com/confidential-containers/containerd/tree/CC-main
• Then we also have to install the kata shim (from their fork: https://github.com/confidential-containers/kata-containers-CCv0). We will let the operator install it by setting preInstall and postInstall images to "" (we handled their work above). So the operator will simply install kata (hopefully won't fail here after we have replaced with the custom containerd).
• Hopefully, after all these are done, we should be able try a workload as described here: https://github.com/confidential-containers/documentation/blob/main/quickstart.md#running-a-workload (unless there are other steps we missed)

[mudler]

Some notes for manual steps so far:

ln -s /etc/systemd/system/k3s.service /etc/systemd/system/containerd.service

# To note, take k8s 1.26.x as containerd version have to be compatible with the version
# provided by intel
# Install the operator as in docs
systemctl stop k3s
cp /opt/confidential-containers/bin/containerd /var/lib/rancher/k3s/data/current/bin/containerd
### After the operator runs, it generate a config.toml for kata
# The original template is here: https://github.com/k3s-io/k3s/blob/master/pkg/agent/templates/templates_linux.go#L10
cat /etc/containerd/config.toml >> /var/lib/rancher/k3s/agent/etc/containerd/config.toml

[jimmykarily]

There is one part of the k3s containerd config that we copy which makes the custom containerd break:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

(if I remove this from /etc/containerd/config.toml Pods come up just fine).
When that's in place, Pods refuse to come up with errors like the one described here: containerd/containerd#4857

Some usefule links:

• "K3s only manages the driver when using the packaged containerd."
• https://github.com/k3s-io/k3s/blob/9980504196ce0cb53c8e04756598d6f8982a5756/pkg/agent/containerd/config_linux.go#L67

What I don't understand is how can it be that k3s kubelet flags don't match the config k3s generated (which is the one we copied). Maybe when we set K3S_CONFIG_FILE this overrides all other config? Maybe k3s was setting kubelet cgroup driver to systemd (that's why it's in the generated config for containerd) and by setting a config file, we removed that option? If that's true, I wonder what other options we may have removed from k3s.

[jimmykarily]

@mauromorales let's remember that forcing the systemd cgroup driver might not work on alpine: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver

When [systemd](https://www.freedesktop.org/wiki/Software/systemd/) is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager.

[jimmykarily]

I summarized all the steps to reproduce what we achieved so far:

Steps to deploy coco on kairos

• Deploy a kairos cluster from latest master (because we need the immucore fix to mount /etc)
  A config like this should be used (see the bundles section):

#cloud-config
bundles:
    - targets:
        - run://quay.io/kairos/community-bundles:system-upgrade-controller_latest
        - run://quay.io/kairos/community-bundles:cert-manager_latest
        - run://quay.io/kairos/community-bundles:kairos_latest
        - run://ttl.sh/kairos-testing/enclave-cc:8h

install:
    auto: true
    device: auto
    reboot: true

k3s:
    enabled: true

users:
    - name: kairos
      passwd: kairos

The enclave-cc bundle is built from this directory: https://github.com/kairos-io/community-bundles/tree/1114-enclave-cc/coco
running this command:

  docker build -t ttl.sh/kairos-testing/enclave-cc:8h . && docker push ttl.sh/kairos-testing/enclave-cc:8h

(We need to package it an ship it)

• A reboot is needed in order for k3s and containerd to be restarted with the new settings (or manually killing containers with k3s-killall.sh and systemctl restart containerd k3s might do)

• Label our node:

  kubectl label --overwrite node $(kubectl get nodes -o jsonpath='{.items[].metadata.name}') node-role.kubernetes.io/worker=""

• Deploy the operator

  kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.4.0

• [Deploy the ccruntime resource]

  kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref=v0.4.0

(wait until they are all running: kubectl get pods -n confidential-containers-system --watch)

• [Temporarily] From the kairos node, add the kata sections at the bottom of
  /etc/containerd/config.toml to /opt/containerd/config.toml

  When we bump immucore and move back to /etc from /opt (See relevant issue) we may not need this step at all. The coco operator will replace our config.toml and this may or may not work because the one we used was generated by k3s and the values in there may be needed.
  UPDATE: After going back to using /etc, the operator simply append the kata plugin settings in the existing config.toml. All good.

• Deploy a workload

  The last part with the verification will only work from within a Pod because the IP address is internal:

  ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")

  You can create a Pod like this:

  apiVersion: v1
  kind: Pod
  metadata:
    name: kubectl
  spec:
    containers:
    - name: kubectl
      image: opensuse/leap
      command: ["/bin/sh", "-ec", "trap : TERM INT; sleep infinity & wait"]
  

  get a shell to it and run the verification commands

  (you will need to install ssh and find out the IP address of the service outside the Pod)

[jimmykarily]

Will wait for feedback from the enclave-cc team and if all is ok, will document this and close the issue.