How to authorize che to create a workspace on amazon EKS

[ghost]

Summary

Previously I was running che in kind for development/testing. I had set the oidc-issuer-url to my application server both in che's configuration and in the kind configuration, and it was working.

For production I try to deploy che to an amazon EKS cluster, with the same oidc-issuer-url, but when making the API request to the dashboard to create a workspace, it returns 403 Forbidden. The logs in the dashboard say devworkspaces.workspace.devfile.io is forbidden: User "<oidc-issuer-url>#e5fb4c6c-1c31-4326-9f09-afcf41e1cda3" cannot create resource "devworkspaces" in API group "workspace.devfile.io" in the namespace "user-che".

I suspect this is more of an AWS issue than a Che issue, but I thought this would be a good place to ask if anyone else has run into the same situation and figured it out.

Relevant information

I tried to follow this guide to bind che's service accounts to an IAM role with all permissions over everything, but still got this 403. I also tried using RBAC, creating a clusterrole with all permissions over everything and a clsuterrolebinding that binds it to al service accounts in the eclipse-che or devworkspace-controller namespaces (since I'm not sure which one actually needs it), but still 403.

This is the clusterrole I used:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: che-permissions
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

And this is the clusterrolebinding I used:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: che-permissions-binding
subjects:
- kind: ServiceAccount
  name: che-dashboard
  namespace: eclipse-che
- kind: ServiceAccount
  name: che
  namespace: eclipse-che
- kind: ServiceAccount
  name: che-gateway
  namespace: eclipse-che
- kind: ServiceAccount
  name: che-service-account
  namespace: eclipse-che
- kind: ServiceAccount
  name: che-operator
  namespace: eclipse-che
- kind: ServiceAccount
  name: default
  namespace: eclipse-che
- kind: ServiceAccount
  name: devworkspace-controller-service-account
  namespace: devworkspace-controller
- kind: ServiceAccount
  name: devworkspace-controller-webhook-server
  namespace: devworkspace-controller
- kind: ServiceAccount
  name: default
  namespace: devworkspace-controller
roleRef:
  kind: ClusterRole
  name: che-permissions
  apiGroup: rbac.authorization.k8s.io

[ibuziuk]

This blog post should be useful - https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html
cc: @karatkep

[tolusha]

As far as I understood, you were able to log in, but you aren't able to create a workspace.
Try to inspect user's permissions then kubectl describe rolebindings -n <USER_NAMESPACE>

[ghost]

For anyone else, we eventually solved this issue. Adding permissions to Che like I was doing didn't help because Che doesn't use its own permissions to create the workspace, it uses the user's permissions. Our problem was that the cluster and Che did not agree on which JWT claim to use to identify users. What we did was in the amazon EKS clsuter settings -> Authentication -> associate identity provider, set the "username claim" to email. Then in the Che configuration, we set the undocumented property CHE_OIDC_USERNAME__CLAIM like this:

spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: email

[tolusha]

Thank you for sharing the information!!!

[venkatesh0822]

For anyone else, we eventually solved this issue. Adding permissions to Che like I was doing didn't help because Che doesn't use its own permissions to create the workspace, it uses the user's permissions. Our problem was that the cluster and Che did not agree on which JWT claim to use to identify users. What we did was in the amazon EKS clsuter settings -> Authentication -> associate identity provider, set the "username claim" to email. Then in the Che configuration, we set the undocumented property CHE_OIDC_USERNAME__CLAIM like this:

spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: email

I have set "username claim" to email in AWS and 'CHE_OIDC_USERNAME__CLAIM' in cr-path.yaml file and deployed che with command 'chectl server:deploy --platform k8s --che-operator-cr-patch-yaml cr-patch.yaml --domain --skip-oidc-provider-check'. But still getting same 403 errror.
Can I know in which che configuration file we need to set 'CHE_OIDC_USERNAME__CLAIM'