Unauthorized when opening the IDE

[l0rd]

Describe the bug

This happened on sandbox this morning

Che version

7.70@latest

Steps to reproduce

Open https://github.com/eclipse-che-demo-app/che-demo-app on sandbox

Expected behavior

No error

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

Linux

Eclipse Che Logs

No response

Additional context

No response

[ibuziuk]

similar 404 error that was caught by periodic tests - output.webm

[dkwon17]

I see similar issues (also similar to #22352) when refreshing the dashboard

This error is quite difficult to reproduce. On some days, I'm able to reproduce this issue very occasionally on the dogfooding cluster, but on other days, I cannot reproduce it at all (I've used a script to refresh the dashboard on dogfooding about 4000 times, and could not reproduce it). I find it much easier to reproduce this error on Dev Sandbox, more specifically the m2 cluster.

I've noticed that these types of errors often occur because some requests made by the dashboard occasionally returns a 401 or 403 error.

403 error

Sometimes the request to get the user namespace results in a 403 error:

This causes other requests like events and pods to fail as well, because they both depend on the value of the user namespace.

In this response header, there are some differences compared to when the namespace request returns 200

The Gap-Auth and Gap-Upstream-Address headers seems to be set by oauth-proxy container in the che-gateway pod:
https://github.com/openshift/oauth-proxy/blob/55e0cd172625b3419c698014fa233e54021af9a3/oauthproxy.go#L99
https://github.com/openshift/oauth-proxy/blob/55e0cd172625b3419c698014fa233e54021af9a3/oauthproxy.go#L830-L834

401 error

In this case, the namespace request succeeded, but for whatever reason, the events and pods request failed with 401. Which is strange because the devworkspace request succeeded.

Just like the 403 case, the Gap-Auth and Gap-Upstream-Address headers are missing in the response header.

[dkwon17]

For the 401/403 error that sometimes appears when refreshing the dashboard, it seems to happen on these requests made by the dashboard:

Request	Fulfilled By
namespace/provision	Che Server
namespace	Che Server
events	Che Dashboard backend
devworkspacetemplates	Che Dashboard backend
devworkspaces	Che Dashboard backend
pods	Che Dashboard backend

For this specific example of the error:

On the screenshot we see the following error code from the response:

FST_UNAUTHORIZED

which is coming from the dashboard backend here, meaning that the authorization header which contains the bearer token is missing.

It's strange that in the screenshot, we see that requests to fetch devworkspacetemplates , devworkspaces , and pods succeeded, but only fetching events failed.

For the events request to reach the dashboard backend api (and same with the che server api), the request needs to

1. Go through the oauth-proxy container (which sets the bearer token in the request header)
2. And then go through the gateway (traefik) container (which uses a custom middleware to set the authorization header with the token set by oauth-proxy) within the che-gateway pod.

Because of oauth-proxy and gateway, the bearer token should be available in the request's header.

But since the request to the dashboard backend ultimately reaches here, it can either mean:

• oauth-proxy container did not set the access token in the request header
• custom traefik middleware was unable to set the authorization header with the access token set by oauth-proxy container
• somehow the authorization header is removed after going through traefik, but before reaching dashboard backend server
• maybe something else

[tolusha]

I am wondering if we can test latest images [1] [2], maybe it will fix our problem.
[1] https://quay.io/openshift/origin-oauth-proxy
[2] https://quay.io/openshift/origin-kube-rbac-proxy

[ibuziuk]

@dkwon17 @tolusha we should probably investigate how ArgoCD manages the OAuth. The UX flow is very similar to what we do with Eclipse Che

[Mbd06b]

I personally encountered this unauthorized issue because I had multiple nodes in my cluster, and not all the nodes had my ODIC issuer arguments applied.

I edited the API Server Configuration in the configuration file for my Kubernetes API server (under /var/snap/microk8s/current/args/kube-apiserver for microk8s).

OIDC Options: Include the following lines (adjusting for your specific Keycloak configuration):

--oidc-issuer-url=https://<KEYCLOAK_DOMAIN>/auth/realms/<REALM_NAME>
--oidc-client-id=<CLIENT_ID>
--oidc-username-claim=email
--oidc-groups-claim=groups

While I ran this from the node I built the cluster from, I discovered that I needed to apply these --odic args on each node, and restart the node for the new configuration to apply, and then my auth issues with login and the devworkspace were resolved.
.
sudo microk8s stop && sudo microk8s start

Just for your consideration if there's a similar multi-node situation to consider.

Linking back to documentation:
#21378 MicroK8S OIDC Issue
#22392 Document how to install Eclipse Che on major k8s providers

[ibuziuk]

Retry logic has been added to the dashboard, and the issue does not seem to be reproducible on nightly. The plan is to promote 3.10 RC changes to staging cluster and verify it properly before closing the issue.

[ibuziuk]

Closing the issue since we need to include it in the 7.76.0 release notes. @dkwon17 please, let me know if we need to proceed differently.