Failed to run the workspace: mkdir /plugins/sidecars: permission denied

[davidwindell]

Describe the bug

I am trying to get Che 7.0.0 multiuser running on K8's, deployed via Rancher using the Helm charts in the repo. I've managed to get everything running now so far as the Che dashboard appears. I am however now stuck from going any further as, when starting a workspace I get the following error:

Error: Failed to run the workspace: "Plugins installation process failed. Error: Plugin broking process for workspace workspacehe0g3fhudssx2s7n failed with error: mkdir /plugins/sidecars: permission denied"

The first thing I did was to check the physical volume, it is just a local folder path on the node at the moment for testing purposes. Here is it's contents:

root@staging1:/test15# ls -la
total 12
drwxrwxrwx  3 root root 4096 Aug 24 11:47 .
drwxr-xr-x 25 root root 4096 Aug 24 11:47 ..
drwxr-xr-x  4 1724 root 4096 Aug 24 11:47 workspacehe0g3fhudssx2s7n

So it looks like Che has been able to create the workspace's folder in the volume without any permission errors. It has also created the plugins folder:

root@staging1:/test15/workspacehe0g3fhudssx2s7n# ls -la
total 16
drwxr-xr-x 4 1724 root 4096 Aug 24 11:55 .
drwxrwxrwx 3 root root 4096 Aug 24 11:55 ..
drwxrwxrwx 2 1724 root 4096 Aug 24 11:55 che-logs-che-plugin-broker
drwxrwxrwx 2 1724 root 4096 Aug 24 11:55 plugins

So...I'm at a loss as to why the plugins/sidecars can't be create. I've tried deleting all folders and starting again but end up with the same problem.

I'd really appreciate any help or pointers on this, it's stopped us moving to the new Che for quite a while now and I'm at a loss.

Thanks!

[ibuziuk]

@amisevsk could you please take a look at this one? What could potentially cause permission denied issues?

Plugins installation process failed. Error: Plugin broking process for workspace workspacehe0g3fhudssx2s7n failed with error: mkdir /plugins/sidecars: permission denied

[amisevsk]

Yeah I'm kind of at a loss here, AFAIK a folder that is 777 shouldn't cause permissions issues.

What is the backing storage for the volumes? What is the mount point in the plugin broker during workspace startup? (note the broker is a short-lived pod during workspace startup, it might be hard to catch its config)

Can you start up a pod that mounts this volume and try modifying the /plugins/ folder? E.g. start up a pod with tail -f /dev/null, mount the volume, and look at it in a shell?

[davidwindell]

Thanks @amisevsk, could you give me a pointer on how to start the broker pod manually? Do I just look for recently stopped containers and then try and run the one that looks like the broker with a custom entrypoint?

[amisevsk]

@davidwindell Without modifying Che that will be hard -- it gets deleted once it's no longer needed by Che. The way I usually debug filesystem issues on k8s/OpenShift is to start up an entirely separate pod that mounts the relevant volume.

E.g. you could use:
pvc-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pvc-test
spec:
  volumes:
    - name: pvc-test
      persistentVolumeClaim:
        claimName: claim-che-workspace
  containers:
    - name: pvc-test
      image: centos
      volumeMounts:
        - mountPath: "/workspaces"
          name: pvc-test
      command: ["tail"]
      args: ["-f", "/dev/null"]

# [in che workspaces namespace]
kubectl apply -f pvc-test.yaml
kubectl exec pvc-test -i -t -- /bin/bash
[check file perms]
kubectl delete po pvc-test

If there are no problems here, we can look into debugging more directly.

[davidwindell]

@amisevsk thanks for the guidance, I launched the pod and everything looks fine with permissions as far as I can tell, I was able to create the sidecars directory as the root user

Would you like access to our cluster to debug this directly?

Thanks!

[davidwindell]

I was able to inspect the broker pod before it stopped and there is a different in the way the broker and test pod show their mounted volumes - could this be related?

Test pod:

 "HostConfig": {
            "Binds": [
                "/test15:/workspaces",

Broker pod:

"HostConfig": {
            "Binds": [
                "/var/lib/kubelet/pods/642d7890-ca8d-11e9-ade1-060ccecb2836/volume-subpaths/test15/eclipse-che-unified-plugin-broker-v0-19/1:/plugins",

[davidwindell]

So, I found this #12661 (comment) and updated my config map to:

CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER: "0" 
CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP: "0" 
CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: "false"

That's 0 (root?) instead of the default 1724, and it looks like the workspaces are starting to run now. I've never been so happy to see a welcome screen before :)

Is setting these values a known issue with vanilla k8s (which Rancher effectively is)?

[dmytro-ndp]

The same error was faced when executing Happy path tests on PR check against K8S che-theia repo:

"SEVERE" "" "http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js 0:7937 Error: Failed to run the workspace: "Plugins installation process failed. Error: Plugin broking process for workspace workspace3xlicbrzx0qupqsi failed with error: mkdir /plugins/sidecars: permission denied"
    at http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:10668
    at http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:20567
    at http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:2066
    at Array.forEach (<anonymous>)
    at e.processNotification (http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:2046)
    at e.processResponse (http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:1911)
    at http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:1214
    at http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:15185
    at Array.forEach (<anonymous>)
    at e.callHandlers (http://che-che.10.0.149.125.nip.io/workspace-loader/loader/bundle.js:1:15158)"

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/Che-Theia-PR-check-E2E-Happy-path-tests-against-K8S/42/artifact/che/e2e/report/Validation_of_workspace_start_Wait_workspace_running_state/browserlogs-Wait_workspace_running_state.txt

[amisevsk]

@davidwindell I'm very sorry for leaving unattended for so long -- I got caught up in some other issues. To be honest, this is behaviour I've not encountered when using Kubernetes. I suspect the issue is caused by using hostPath volumes, as the docs warn

the files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume

There's probably a way to get non-root Che to work with hostPath volumes, but that beyond my knowledge of Kubernetes at the moment. Alternatively, using a different storage class may resolve the problem -- I'm not enough of a Kubernetes admin to say for sure.

The general reason we're seeing this issue is that running on OpenShift requires containers to not run as root, so Che is designed with this assumption in mind by default. I didn't think, until digging into this, that there was a default configuration of k8s that only works if containers run as root, since this practice is generally advised against.

[amisevsk]

@davidwindell If you are still interested in poking at this one (I understand if you're not or don't have time) could you re-try the manual broker startup with updated pvc-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pvc-test
spec:
  volumes:
    - name: pvc-test
      persistentVolumeClaim:
        claimName: claim-che-workspace
  containers:
    - name: pvc-test
      image: centos
      volumeMounts:
        - mountPath: "/workspaces"
          name: pvc-test
      command: ["tail"]
      args: ["-f", "/dev/null"]
  securityContext:
    runAsUser: 1724
    fsGroup: 1724

This is more similar to what the broker actually runs as, so it might make the issue clearer (afaik user 1724 should be able to write /plugins, which is owned by 1724).

[davidwindell]

Thanks @amisevsk, I'm going to try and use an EBS volume and will report back in a few days.

[bryantson]

Facing the similar issue and tried:

CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER: "0"
CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP: "0"

It still does not work for me.

[davidwindell]

Can confirm, the problem is isolated to hostPath volumes (which I was using for testing).

Switching to EBS finally solves this issue for me. Thank you @amisevsk for your help with this.

[gorshkov-leonid]

Hello. Does somebody know how to fix the same problem for case of docker-desktop?
I use
chectl server:start --platform=docker-desktop --cheimage=eclipse/che-server:7.7.0. Server starts successfully, but during start of workspace I can see the same error.

Of cause, It is Windows.