Allow DevWorkspace CR to specify pod spec overrides

[amisevsk]

Which area this feature is related to?

/area devworkspace

Which functionality do you think we should add?

Add a field to the DevWorkspace CR spec that allows overriding fields in the k8s Deployment's pod template, e.g.

 kind: DevWorkspace
 apiVersion: workspace.devfile.io/v1alpha2
 metadata:
   name: myspacework
 spec:
   started: true
+  pods:
+    metadata:
+      annotations:
+        io.openshift.userns: "true"
+        io.kubernetes.cri-o.userns-mode: "auto:size=65536;map-to-root=true"  # <-- user namespace
+        openshift.io/scc: container-build
+    spec:
+      runtimeClassName: kata
+      schedulerName: stork      
   template:
   (...)

This field would not be included in the Devfile API

Why is this needed? Is your feature request related to a problem?

There are a variety of cases where overriding specific Pod fields is required by end-users. For example, a user may want to

• Apply specific annotations to configure how their DevWorkspace runs
• Override specific pod fields such as runtimeClassName and schedulerName

In general, there's a wide variety of potential things someone may need to override to make DevWorkspaces work for them, but each individual case is not common enough that it fits as part of the dedicated API. Up until now, we generally add special-case handling for individual fields (e.g. SCCs), but this is not flexible enough to cover all use cases, and complicates set up significantly.

Describe the solution you'd like

As above, add field pods to the DevWorkspace API. The contents of the pods field would be corev1.PodSpec. Any value there would be strategic-merged into the resulting Pod spec generated from the DevWorkspace's spec.

Describe alternatives you've considered

This could also be implemented using free-form attributes, but this would mean

• It's harder to use when writing/editing a DevWorkspace by hand
• There is no validation or documentation on the field
• It's potentially more error-prone

Additional context

DevWorkspaceOperator issue: devfile/devworkspace-operator#852

[elsony]

If someone specifies the kubernetes component and provides an inline or link to kubernetes yaml file, the pod settings may conflict with these metadata. How are we planning to resolve that?

[amisevsk]

If someone specifies the kubernetes component and provides an inline or link to kubernetes yaml file, the pod settings may conflict with these metadata. How are we planning to resolve that?

To clarify, the new pods field is intended to only apply to the deployment/pod that is created to store components of the container type, not anything else. Perhaps it is better to name it pod?

Any specified Kubernetes component is implicitly not a part of the core devworkspace deployment. If e.g. a devfile specifies a k8s pod in a Kubernetes component, that will generally have to be provisioned as a standalone ("dedicated") pod rather than merged into the other container components. The podSpec field only overrides settings for the components that cannot be configured directly (the deployment/pod that contains container components only).

The documentation for Kubernetes components states their purpose is

Allows importing into the devworkspace the Kubernetes resources defined in a given manifest. For example this allows reusing the Kubernetes definitions used to deploy some runtime components in production.

If we do some processing (e.g. to include something defined in a k8s component in the same deployment as container components) then we would be prone to breaking that use case. Examples of this would be

• Conflicting configuration between what's needed for container components and the explicit pod
• Port/volume collisions between the k8s and container components.

[l0rd]

@amisevsk I named it pods in devfile/devworkspace-operator#852 because I initially thought that we should apply the spec to every deployment/pod, not just the main one. I am thinking in particular about annotations, labels, schedulers: if the DevWorkspace includes more than one pod, it makes sense to apply it to all of them. But to be honest we don't have any real use case for it, those are just my speculation and I am perfectly fine to just override the main podSpec. But yes, we should probably rename it pod, mainPod or devPod to avoid confusions.

[amisevsk]

@l0rd I suppose it would make sense to apply the spec to both the main deployment's pods and also any dedicated pods specified -- in that sense this field would apply to all container components in the spec. To my knowledge, dedicatedPod isn't used currently, which is why I didn't think of it. This would mean keeping the pods name rather than pod.

Other pods, started as e.g. part of kubernetes components, could specify their full specs in the inlined/referenced yaml so I don't think the override is necessary there.

I'm still not entirely sure on naming though -- I prefer pods to pod/mainPod/devPod, but maybe something like podSpecOverride is better for the sake of being explicit?

[l0rd]

I like podSpecOverride more and I would override kubernetes components too but that can be discussed later as that's just a philosophical exercise at the moment.