New blog post about Che deployment on AKS

_posts/2022-07-25-installing-eclipse-che-on-aks.adoc

@@ -0,0 +1,195 @@
+---
+title: Installing Eclipse Che on the Azure Kubernetes Service (AKS)
+layout: post
+author: Serguei Gorokhov and Piotr Karatkevich
+description: >-
+  In this blog post, we'll tell the story how we installed Eclipse Che on Microsoft Azure, what challenges we met and how solved them.
+categories: []
+keywords: ['AKS', 'Azure Kubernetes Service', 'AAD', 'Azure Active Directory']
+slug: /@karatkep/installing-eclipse-che-on-aks
+---
+
+We installed {prod} on AKS (Azure Kubernetes Service) cluster integrated with AAD (Azure Active Directory) according to link:https://www.eclipse.org/che/docs/che-7/installation-guide/installing-che-on-microsoft-azure/[instruction]. And to our surprise it did not work. Here at Epam Systems we love complex challenges, so we decided not to stop and deep dived into {prod} world to improve interaction with Microsoft Azure.
+
+NOTE: Taking into account that {prod} calls {kubernetes} API server using user's identity tokens generated by Identity Provider, AKS needs to be integrated with AAD, otherwise AKS will reject the token. Details how to integrate AKS with AAD can be found link:https://docs.microsoft.com/en-us/azure/aks/managed-aad[here].
+
+Architecture diagram below can be used to get a context about our case. Please note that some of the components are omitted for the sake of simplicity. But it should be enough to get up to speed and dive into {prod} world together with us.
+
+.Eclipse Che components deployed on AKS
+image::/assets/img/installing-eclipse-che-on-aks/che-in-aks.png[Eclipse Che components deployed on AKS]
+
+== Challenge 1: AKS does not accept id_token
+OAuth2 Proxy is configured to send `id_token` to Traefik for installations made for {kubernetes}. Traefik, in its turn, sends `id_token` to all upsteams, including such components like Che server and User Dashboard. Che server and User Dashboard call AKS API Server using provided `id_token`. But in case of AAD integration enabled for AKS, AKS API Server does not accept `id_token` (it only accepts `access_token`).
+
+=== How did we solve the first challenge
+Obviously we need a way to ask OAuth2 Proxy to send `access_token` instead of `id_token`. But OAuth2 Proxy version before alpha configuration does not support token replacement. It can be configured to pass `access_token` via X-Forwarded-Access-Token header, but not to replace `id_token` in Authorization header. So, first step is to add `access_token` via X-Forwarded-Access-Token header:
+
+[source]
+----
+pass_access_token = true
+----
+
+Then we need somehow to replace Authorization header on X-Forwarded-Access-Token header. Fortunately, we have Traefik in place. So, let's ask Traefik to make a necessary replacement:
+[source,yaml]
+----
+http:
+  middlewares:
+    che-header-rewrite:
+      plugin:
+        header-rewrite:
+          from: X-Forwarded-Access-Token
+          prefix: 'Bearer '
+          to: Authorization
+----
+
+And voila, we have it workable, like a magic. But what's about others who will also install {prod} on AKS? We already mentioned, that at Epam Systems we love complex challenges. And of course we love coding. So, we opened the link:https://github.com/eclipse/che/issues/21450[che-operator enhancements] and proposed the link:https://github.com/eclipse-che/che-operator/pull/1400[improvements].
+
+In consequence of proposed improvements new configuration field `identityToken` has been introduced into {prod} starting version 7.50+. And user can easily configure what token to be passed to upstreams in CheCluster CR (Custom Resource):
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      identityToken: access_token
+----
+There are two types of tokens supported: `id_token` and `access_token`. Default value is `id_token`.
+
+NOTE: The field `identityToken` is specific to {prod} installations made for Kubernetes only and ignored for OpenShift.
+
+== Challenge 2: No way to obtain proper access token accepted by AKS
+This is a continuation of Challenge 1 in a way that after we taught {prod} to leverage `access_token` instead of `id_token` to access AKS, we need to make sure this `access_token` is accepted by AKS.
+
+When AKS integration with AAD is enabled, AKS expects OpenID Connect (OIDC) tokens issued to the certain application, pre-registered in AAD: 'Azure Kubernetes Service AAD Server'.
+
+.Azure Kubernetes Service AAD Server application
+image::/assets/img/installing-eclipse-che-on-aks/aks-aad-server-app.png[Azure Kubernetes Service AAD Server]
+
+If we leave default settings of OAuth2 Proxy, leveraged in Gateway, it uses standard set of OIDC scopes and for them AAD returns an `access_token` that can be used against Microsoft Graph, not AKS.
+
+In order to obtain a token suitable for AKS, additional scope should be configured for OAuth2 Proxy, representing Azure Kubernetes Service AAD Server application:
+[source,yaml]
+----
+6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+----
+where GUID is a Application ID for AKS registered in AAD.
+
+=== How did we solve the second challenge
+New configuration field `oAuthScope` has been introduced into {prod} starting version 7.50+ in scope of already opened link:https://github.com/eclipse/che/issues/21450[che-operator enhancements] and provided link:https://github.com/eclipse-che/che-operator/pull/1400[improvements]. So user can easily configure authorization scopes in CheCluster CR:
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+----
+NOTE: The field `identityToken` is specific to {prod} installations made for {kubernetes} only and ignored for OpenShift.
+
+== Challenge 3: Che server requires mandatory 'email' claim which is absent in AKS access_token
+Besides passing OIDC token to AKS for authorization purposes, Che server also uses it for creating a 'user' record in PostgreSQL database. There are two pieces of information (claims) about current user that Che server expects to extract from the token: name and email. Che allows configuring of user name claim via CheCluster CR in AKS (`CHE_OIDC_USERNAME__CLAIM` according to link:https://www.eclipse.org/che/docs/next/administration-guide/advanced-configuration-options-for-the-che-server-component/#_che_oidc_username_claim[documentation]), but email claim name is hardcoded to `email` and cannot be changed. This hardcode becomes a problem for AKS setup since `access_token` returned by AAD for AKS does not contain `email` claim (see current structure below), and customization is not possible here since AKS application registration in AAD is maintained by Microsoft.
+[source,jwt]
+----
+{
+  "typ": "JWT",
+  "alg": "RS256",
+  "x5t": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI",
+  "kid": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI"
+}.{
+  "aud": "6dae42f8-4368-4678-94ff-3960e28e3630",
+  "iss": "https://sts.windows.net/f974903a-f693-4ca2-aa63-45e5bd2d1318/",
+  "iat": 1656412511,
+  "nbf": 1656412511,
+  "exp": 1656417802,
+  "acr": "1",
+  "aio": "AUQAu/8TAABB1o0F/h6L8SJWLibrQqPKrVF7qpdIFiQ0qeIj/wNUfcLkM87v+wgvQ+I8uj0WC2Y6WEoHlBcsJdt0m+kJ4dFeNw==",
+  "amr": [
+    "pwd"
+  ],
+  "appid": "1c0b88a0-8ec2-4cca-a2b0-4f468110ef86",
+  "appidacr": "1",
+  "family_name": "Kluklu",
+  "given_name": "Tratata",
+  "groups": [
+    "73fdccdf-e10d-45f9-b7f6-31848842999f",
+    "5988d043-3af9-4e81-b041-90b3456f9f4e",
+    "bddcb049-0337-4dec-bbaf-7600b8c12623"
+  ],
+  "ipaddr": "10.123.51.3",
+  "name": "Tratata Kluklu",
+  "oid": "2cf0521c-c76d-4e7c-b41f-863674057db3",
+  "onprem_sid": "S-2-4-31-6364504-298352422-13854118-387761",
+  "puid": "3213CDFA3CAF2IA5",
+  "rh": "0.AQkA1NIbtJ39JkuKaqbJ82fCHscCrm1oG3hTlP47YOQHDjAJAHg.",
+  "scp": "user.read",
+  "sub": "qweBLvHX49QA5WlXpJzq_erXQ2NldnSqpgY93oALLDY",
+  "tid": "a385e78a-aedc-4033-82ba-e6ef88120591",
+  "unique_name": "Tratata.Kluklu@gmail.com",
+  "upn": "Tratata.Kluklu@gmail.com",
+  "uti": "lfZmPsgcWmS3dG78GpMjRA",
+  "ver": "1.0",
+  "wids": [
+    "c79abafb-610b-4a34-82e2-ef7a293db6ca"
+  ]
+}.[Signature]
+----
+
+
+=== How did we solve the third challenge
+The same as above, we need some enhancements on {prod} side to allow user to configure what token claim need to be used to extract user email. As we did it before, we opened the link:https://github.com/eclipse/che/issues/21515[che-server enhancement] and proposed the link:https://github.com/eclipse-che/che-server/pull/324[improvement].
+
+Now user can configure email claim to be used when parsing JWT token:
+[source,yaml]
+----
+spec:
+  components:
+    cheServer:
+      extraProperties:
+        CHE_OIDC_EMAIL__CLAIM: unique_name
+----
+If not defined, the fallback value is `email`.
+
+== Conclusion
+In this post, we talked about the challenges we met during {prod} installation on AKS and explained how to solve them using the latest versions of the product.
+
+Now, user has all needed things configurable to be able to run successfully {prod} on AKS. For example, in our particular case we prepared yaml file that overrides the default values in CheCluster CR.
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
+      identityToken: access_token
+      oAuthClientName: {CLIENT_ID}
+      oAuthSecret: {CLIENT_SECRET}
+      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+  components:
+    cheServer:
+      extraProperties:
+        CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{TENANT_ID}/v2.0/
+        CHE_OIDC_EMAIL__CLAIM: unique_name 
+----
+
+* `TENANT_ID` - Directory (tenant) ID, see Figure 3.
+* `CLIENT_ID` - Application (client) ID, see Figure 3.
+* `CLIENT_SECRET` - Client secret, you can manage it in 'Certificates & secret' section
+
+.Registered Che application
+image::/assets/img/installing-eclipse-che-on-aks/azure-che-demo-app.png[Registered Che application]
+
+WARNING: Don't forget to configure API permissions to authorize your application to call AKS Server API.
+
+.AKS API permissions
+image::/assets/img/installing-eclipse-che-on-aks/aks-api-permissions.png[AKS API permissions]
+
+After all preparations are completed, `chectl` deploy command can be used to install {prod} on AKS using yaml file above:
+[source,shell]
+----
+chectl server:deploy \
+       --platform=k8s \
+       --installer=operator \
+       --che-operator-cr-patch-yaml=che.yaml \
+       --skip-oidc-provider-check \
+       --skip-cert-manager \
+       --domain=eclipse-che-demo.mydomain.com
+----
+
+NOTE: In our case we already configured `cert-manager` and created `domain` according to the link:https://www.eclipse.org/che/docs/che-7/installation-guide/installing-che-on-microsoft-azure/[instruction].