Add support for overriding any pod spec field in DevWorkspace deployment

[amisevsk]

What does this PR do?

Add support for specifying Pod spec overrides in DevWorkspaces. This PR is open as a draft for review/discussion on approach.

The first commit in this PR updates the devfile/api dependency to point at my fork, to pull in changes from PR devfile/api#872. Before merging this PR, the devfile/api PR should be merged and the devfile/api dependency should be updated normally.

What issues does this PR fix or reference?

Related: devfile/api#860
Closes #852

Is it tested? How?

To test the changes, there's a new sample:

oc apply -f samples/pod-spec-overrides.yaml

This workspace will likely fail to start as it sets the scheduler to stork and the runtimeClassName to kata, which are likely not available. To verify things are working as expected, check that fields are overridden in the Deployment for that workspace

PR Checklist

• E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
  • v8-devworkspace-operator-e2e: DevWorkspace e2e test
  • v8-che-happy-path: Happy path for verification integration with Che

[amisevsk]

I've pushed fixes to take metadata into account as well -- it turns out that controller-gen is unable to generate embedded metadata within a CRD so we have to copy the struct ourselves -- see devfile/api@dfc7f70.

I've just tested again and it appears that everything works as intended. A container image from the current changes is built and pushed to quay.io/amisevsk/devworkspace-controller:pod-spec-overrides

To test it on a cluster, run the following:

export NAMESPACE=dw #or your choice
export DWO_IMG=quay.io/amisevsk/devworkspace-controller:pod-spec-overrides
make install

[amisevsk]

PRs in devfile/api are merged and this PR is ready for review.

Note one issue that is a side-effect of this PR is that the DevWorkspace and DevWorkspaceTemplate CRDs are now too big to apply using kubectl apply. This is because using client-side apply adds the kubectl.kubernetes.io/last-applied-configuration containing the full json of the object applied, and the new CRD instances are larger than the max length of an annotation. This fails with message

The CustomResourceDefinition "devworkspaces.workspace.devfile.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

To avoid this issue, the make install rule now uses server-side apply instead, which does not create the annotation and instead sets .metadata.managedFields directly. However, the drawback here is that we also have to use the --force-conflicts parameter: since the inject-cabundle annotation is present on the CRDs, the caBundle field is owned by Go-http-client, which injects the bundle. If CRDs are already present, server-side apply would fail with

Apply failed with 1 conflict: conflict with "Go-http-client" using apiextensions.k8s.io/v1: .spec.conversion.webhook.clientConfig.caBundle

Before merging this PR, it will be necessary to update documentation on applying generated yaml files directly and verify that tools which apply them can accomodate the change.

[amisevsk]

To test these changes via OLM, I've build an index image quay.io/amisevsk/devworkspace-operator-index:pod-spec-overrides this can be used to create a catalogsource on the cluster and deploy the current changes via OLM.

[AObuchow]

Code looks good to me (I love how elegant the actual implementation is) 👍
I ended up testing this on OLM which worked well.

The deployment did fail because the kata RuntimeClass wasn't found, but I could see the workspace deployment was modified as expected:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
  resourceVersion: '52614'
  name: workspace2d70776a4c454fa4
  uid: 0ab3d1e8-e3cf-463f-b954-b637cd8b61e3
  creationTimestamp: '2022-09-22T17:17:00Z'
  generation: 2
 (...)
  namespace: default
  ownerReferences:
    - apiVersion: workspace.devfile.io/v1alpha2
      kind: DevWorkspace
      name: theia-next
      uid: 2d70776a-4c45-4fa4-b5bb-c7c683ba5110
      controller: true
      blockOwnerDeletion: true
  labels:
    controller.devfile.io/creator: ''
    controller.devfile.io/devworkspace_id: workspace2d70776a4c454fa4
    controller.devfile.io/devworkspace_name: theia-next
spec:
  replicas: 0
  selector:
    matchLabels:
      controller.devfile.io/devworkspace_id: workspace2d70776a4c454fa4
  template:
    metadata:
      name: workspace2d70776a4c454fa4
      namespace: default
      creationTimestamp: null
      labels:
        controller.devfile.io/creator: ''
        controller.devfile.io/devworkspace_id: workspace2d70776a4c454fa4
        controller.devfile.io/devworkspace_name: theia-next
      annotations:
+        io.kubernetes.cri-o.userns-mode: 'auto:size=65536;map-to-root=true'
+        io.openshift.userns: 'true'
+        openshift.io/scc: container-build
    spec:
(...)
      serviceAccountName: workspace2d70776a4c454fa4-sa
      imagePullSecrets:
        - name: workspace2d70776a4c454fa4-sa-dockercfg-4wtnn
+      schedulerName: stork
      terminationGracePeriodSeconds: 10
+      runtimeClassName: kata

[dkwon17]

LGTM and works well 👍 (tested with OLM)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisevsk, AObuchow, dkwon17

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [amisevsk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dkwon17]

I did notice though, if try to test by running DWO via:

export NAMESPACE="devworkspace-controller"
make install
oc patch deployment/devworkspace-controller-manager --patch "{\"spec\":{\"replicas\":0}}" -n $NAMESPACE
make run

and then when I run:

oc apply -f samples/pod-spec-overrides.yaml

the spec.podSpecOverride disappears from the DevWorkspace. I don't think this should happen? I didn't experience this when testing with OLM.

I had to add the spec.podSpecOverride content manually after I created the DevWorkspace. After doing that, the pod spec overriding was working

[amisevsk]

Superceded by #944