Kubernetes Support

[MoisesGSalas]

Description

The plugin at the moment does not support kubernetes in any way or form.

In the Docker deployment we use AppArmor to restrict the container's access to certain resources. To enforce the rules of the AppArmor profile on the container we have to make sure that the profile is loaded on the host machine. We accomplish this task by running a job on a privileged container that loads the profile onto the host.

In the case of Kubernetes we face the challenge of loading the AppArmor profile onto all the nodes of the kubernetes cluster (due to the pod that hosts the codejail container possibly being assigned to any node).
Once that is done it is possible to deploy the codejail service using a secured container.

Loading the profile is not even the first blocker, the Amazon Linux images used by default on EKS do not have AppArmor enabled (I don't know what OSes are used on the other providers). One option is to use Ubuntu AMIs for EKS
which already have AppArmor enabled. Once that is taken care of, is possible to load the profile using several strategies. Each one with their own limitations.

I have deployed an EKS cluster using the ami-0a9aad2548662426 AMI from Canonical and tried both, SSHing into the nodes and loading the profile manually and using the DaemonSet, in both cases I was able to successfully load the profile onto the node and ran the codejail service using a deployment that looked like this.

The discussion I want to start is how the Kubernetes support should look from now onwards. The CodeJail service has a series of requirements that make it inherently difficult to run in k8s securely. Ensuring that the plugin setups correctly all these requirements in multiple environments (at least most common ones) seems like a tall order for the time being. What I'm feeling inclined at the moment is to offer the templates for the service/deployment but leave the initialization to the operator and even allow the service to run without the profile enforcement on the container if the user decides to. Maybe I'm not being ingenious enough and if someone has a nicer approach I would love to hear it.

A little summary of the problems we currently have:

• We need to ensure that the nodes have AppArmor enabled.
• We need to load the AppArmor profile onto each node. Using a daemon set seems like the most sensible solution, but running the privileged container appears to be a little out of bounds for the plugin.
  • In addition the DaemonSet can't update or delete profiles. We may avoid this issue if we version the profiles (although I don't see them changing too often)
• Defining the k8s-job for the init task will not work even if it was a noop due to incompatible naming. @angonz made a few fixes here.

[gabor-boros]

@MoisesGSalas Is this still an issue? I was able to use Codejail with tutor-contrib-codejail successfully.

[MoisesGSalas]

I'm glad that you were able to use it @gabor-boros. Considering we have some basic Kubernetes support since 14.1.0 I will close the issue.